Prometheus集群容災(zāi)備份與故障切換方案技術(shù)棧Kubernetes v1.32.13 Rocky Linux 8.6 Prometheus Containerd 1.7.x操作環(huán)境 / 對(duì)接原理 / 詳細(xì)步驟 / 完整命令 / 配置文件 / 驗(yàn)證流程 / 排錯(cuò)方案Prometheus集群容災(zāi)備份與故障切換方案操作環(huán)境K8s 集群 3 節(jié)點(diǎn)k8s-master(192.168.1.10), k8s-node1(192.168.1.11), k8s-node2(192.168.1.12)K8s 版本 v1.32.13監(jiān)控組件Prometheus監(jiān)控已部署對(duì)應(yīng)監(jiān)控組件 PodPrometheus 已配置服務(wù)發(fā)現(xiàn)Grafana 已對(duì)接數(shù)據(jù)源操作系統(tǒng) Rocky Linux 8.6容器運(yùn)行時(shí) Containerd 1.7.x已配置監(jiān)控專(zhuān)用命名空間 monitoring監(jiān)控?cái)?shù)據(jù)持久化已配置 PVC 綁定數(shù)據(jù)留存策略已規(guī)劃監(jiān)控組件資源限制已設(shè)置已配置 RBAC 權(quán)限監(jiān)控組件具備集群資源讀取權(quán)限網(wǎng)絡(luò)策略已放行監(jiān)控通信端口對(duì)接原理Prometheus集群容災(zāi)備份與故障切換方案是 K8s 集群監(jiān)控告警系統(tǒng)運(yùn)維中的核心操作場(chǎng)景。K8s 監(jiān)控體系通過(guò)指標(biāo)采集Exporter、數(shù)據(jù)存儲(chǔ)Prometheus TSDB、可視化展示Grafana、告警管理Alertmanager四大核心組件實(shí)現(xiàn)全鏈路監(jiān)控。Prometheus監(jiān)控作為監(jiān)控體系的具體組件負(fù)責(zé)指標(biāo)采集、數(shù)據(jù)存儲(chǔ)、可視化展示或告警分發(fā)。運(yùn)維操作的核心目標(biāo)是確保監(jiān)控系統(tǒng)的可用性、數(shù)據(jù)質(zhì)量、告警精準(zhǔn)度和性能穩(wěn)定性通過(guò)規(guī)范化的部署配置確保監(jiān)控組件穩(wěn)定運(yùn)行通過(guò)精細(xì)化的指標(biāo)采集確保數(shù)據(jù)完整準(zhǔn)確通過(guò)合理的告警規(guī)則確保故障及時(shí)發(fā)現(xiàn)不誤報(bào)通過(guò)性能調(diào)優(yōu)確保監(jiān)控系統(tǒng)不成為瓶頸通過(guò)高可用和容災(zāi)備份確保監(jiān)控不中斷。所有操作需遵循業(yè)務(wù)無(wú)感知原則對(duì)監(jiān)控系統(tǒng)的變更采用灰度和滾動(dòng)方式避免影響業(yè)務(wù)監(jiān)控。詳細(xì)步驟1. 監(jiān)控現(xiàn)狀盤(pán)點(diǎn)與組件狀態(tài)檢查# 1. 檢查集群節(jié)點(diǎn)狀態(tài) kubectl get nodes -o wide kubectl get pods -n monitoring -o wide 2/dev/null || kubectl get pods -A | grep -E prometheus|grafana|alertmanager|exporter ? # 2. 檢查監(jiān)控組件狀態(tài) kubectl get pods -n monitoring -o wide 2/dev/null kubectl get svc -n monitoring -o wide 2/dev/null kubectl get pvc -n monitoring -o wide 2/dev/null ? # 3. 檢查 Prometheus 狀態(tài) kubectl get pods -n monitoring -l appprometheus -o wide 2/dev/null kubectl logs -n monitoring -l appprometheus --tail30 2/dev/null | tail -20 ? # 4. 檢查 Grafana 狀態(tài) kubectl get pods -n monitoring -l appgrafana -o wide 2/dev/null kubectl logs -n monitoring -l appgrafana --tail20 2/dev/null | tail -10 ? # 5. 檢查 Alertmanager 狀態(tài) kubectl get pods -n monitoring -l appalertmanager -o wide 2/dev/null kubectl logs -n monitoring -l appalertmanager --tail20 2/dev/null | tail -10 ? # 6. 檢查 Exporter 狀態(tài) kubectl get pods -A | grep -E node-exporter|cadvisor|kube-state-metrics kubectl get svc -A | grep -E node-exporter|cadvisor|kube-state-metrics ? # 7. 檢查監(jiān)控?cái)?shù)據(jù)采集狀態(tài) # Prometheus Targets 狀態(tài) # kubectl port-forward -n monitoring svc/prometheus 9090:9090 # curl -s http://localhost:9090/api/v1/targets | jq .data.activeTargets[] | {job, health, lastError} | head -50 ? # 8. 導(dǎo)出監(jiān)控配置備份 kubectl get cm -n monitoring -o yaml /tmp/monitor_cm_backup_$(date %Y%m%d).yaml 2/dev/null kubectl get secret -n monitoring -o yaml /tmp/monitor_secret_backup_$(date %Y%m%d).yaml 2/dev/null kubectl get pvc -n monitoring -o yaml /tmp/monitor_pvc_backup_$(date %Y%m%d).yaml 2/dev/null echo 監(jiān)控配置已備份到 /tmp/ ?2. 制定操作方案與備份防護(hù)# 1. 備份監(jiān)控配置操作前必做 kubectl get cm -n monitoring -o yaml /tmp/monitor_cm_backup_$(date %Y%m%d_%H%M%S).yaml 2/dev/null kubectl get secret -n monitoring -o yaml /tmp/monitor_secret_backup_$(date %Y%m%d_%H%M%S).yaml 2/dev/null kubectl get deployment,statefulset,daemonset -n monitoring -o yaml /tmp/monitor_workload_backup_$(date %Y%m%d_%H%M%S).yaml 2/dev/null kubectl get pvc -n monitoring -o yaml /tmp/monitor_pvc_backup_$(date %Y%m%d_%H%M%S).yaml 2/dev/null ? # 2. 記錄當(dāng)前監(jiān)控狀態(tài) echo 監(jiān)控狀態(tài)記錄 $(date) /tmp/monitor_status_$(date %Y%m%d).txt echo --- 監(jiān)控組件 Pod --- /tmp/monitor_status_$(date %Y%m%d).txt kubectl get pods -n monitoring -o wide 2/dev/null /tmp/monitor_status_$(date %Y%m%d).txt echo --- 監(jiān)控 Service --- /tmp/monitor_status_$(date %Y%m%d).txt kubectl get svc -n monitoring -o wide 2/dev/null /tmp/monitor_status_$(date %Y%m%d).txt echo --- 監(jiān)控 PVC --- /tmp/monitor_status_$(date %Y%m%d).txt kubectl get pvc -n monitoring -o wide 2/dev/null /tmp/monitor_status_$(date %Y%m%d).txt cat /tmp/monitor_status_$(date %Y%m%d).txt ? # 3. 制定操作方案 cat /tmp/monitor_operation_plan.md EOF # 監(jiān)控操作方案 ## 一、操作目標(biāo) ## 二、影響范圍評(píng)估 ## 三、操作步驟與時(shí)間窗口 ## 四、回滾方案 ## 五、驗(yàn)證標(biāo)準(zhǔn) ## 六、風(fēng)險(xiǎn)點(diǎn)與應(yīng)對(duì)措施 EOF echo 操作方案模板已創(chuàng)建 ? # 4. 確認(rèn)業(yè)務(wù)窗口 echo 當(dāng)前時(shí)間: $(date) echo 建議在業(yè)務(wù)低峰期執(zhí)行監(jiān)控操作避免影響業(yè)務(wù)監(jiān)控 ? # 5. 通知相關(guān)業(yè)務(wù)方 # echo 監(jiān)控運(yùn)維操作通知 | mail -s 監(jiān)控運(yùn)維通知 adminexample.com ?3. 執(zhí)行監(jiān)控組件配置操作# 1. 檢查監(jiān)控組件配置 # Prometheus 配置 kubectl get cm -n monitoring prometheus-config -o yaml 2/dev/null | head -80 # Grafana 配置 kubectl get cm -n monitoring grafana-config -o yaml 2/dev/null | head -50 # Alertmanager 配置 kubectl get cm -n monitoring alertmanager-config -o yaml 2/dev/null | head -50 ? # 2. 檢查監(jiān)控組件日志 # Prometheus 日志 kubectl logs -n monitoring -l appprometheus --tail50 2/dev/null | tail -30 # Grafana 日志 kubectl logs -n monitoring -l appgrafana --tail30 2/dev/null | tail -20 # Alertmanager 日志 kubectl logs -n monitoring -l appalertmanager --tail30 2/dev/null | tail -20 ? # 3. 檢查 Prometheus 配置語(yǔ)法 # kubectl exec -n monitoring prometheus-0 -- promtool check config /etc/prometheus/prometheus.yml ? # 4. 檢查 Alertmanager 配置語(yǔ)法 # kubectl exec -n monitoring alertmanager-0 -- amtool check-config /etc/alertmanager/alertmanager.yml ? # 5. 執(zhí)行具體監(jiān)控配置操作根據(jù)標(biāo)題調(diào)整 echo 執(zhí)行監(jiān)控組件具體配置操作... echo 請(qǐng)根據(jù)操作方案執(zhí)行具體步驟 ? # 6. 應(yīng)用監(jiān)控配置變更 # kubectl apply -f /tmp/monitor_config.yaml # kubectl rollout restart deployment/prometheus -n monitoring # kubectl rollout restart deployment/grafana -n monitoring # kubectl rollout restart statefulset/alertmanager -n monitoring ? # 7. 等待監(jiān)控組件重啟完成 kubectl rollout status deployment/prometheus -n monitoring --timeout120s 2/dev/null kubectl rollout status deployment/grafana -n monitoring --timeout120s 2/dev/null kubectl rollout status statefulset/alertmanager -n monitoring --timeout120s 2/dev/null echo 監(jiān)控組件重啟完成 ?4. 驗(yàn)證監(jiān)控?cái)?shù)據(jù)采集與業(yè)務(wù)無(wú)感知# 1. 驗(yàn)證監(jiān)控組件狀態(tài) kubectl get pods -n monitoring -o wide 2/dev/null # 預(yù)期所有監(jiān)控組件 Pod RunningREADY 正常 ? # 2. 驗(yàn)證 Prometheus 數(shù)據(jù)采集 # 檢查 Targets 狀態(tài) # kubectl port-forward -n monitoring svc/prometheus 9090:9090 # sleep 5 # curl -s http://localhost:9090/api/v1/targets | jq .data.activeTargets[] | select(.health!up) | {job, health, lastError} # 預(yù)期所有 Targets 狀態(tài)為 up無(wú)異常 ? # 3. 驗(yàn)證 Prometheus 查詢(xún) # curl -s http://localhost:9090/api/v1/query?queryup | jq .data.result[] | {metric: .metric.job, value: .value[1]} | head -20 # 預(yù)期查詢(xún)返回正常數(shù)據(jù) ? # 4. 驗(yàn)證 Grafana 訪問(wèn) # kubectl port-forward -n monitoring svc/grafana 3000:3000 # sleep 5 # curl -s http://admin:adminlocalhost:3000/api/health # 預(yù)期Grafana 健康檢查正常 ? # 5. 驗(yàn)證 Grafana 數(shù)據(jù)源 # curl -s http://admin:adminlocalhost:3000/api/datasources | jq .[].name # 預(yù)期Prometheus 數(shù)據(jù)源已配置 ? # 6. 驗(yàn)證 Alertmanager 狀態(tài) # kubectl port-forward -n monitoring svc/alertmanager 9093:9093 # sleep 5 # curl -s http://localhost:9093/api/v2/status | jq .cluster # 預(yù)期Alertmanager 集群狀態(tài)正常 ? # 7. 驗(yàn)證告警規(guī)則 # curl -s http://localhost:9090/api/v1/rules | jq .data.groups[] | {name, rules: [.rules[] | {name, health}]} | head -50 # 預(yù)期告警規(guī)則已加載狀態(tài)健康 ? # 8. 清理 port-forward # pkill -f port-forward 2/dev/null echo 驗(yàn)證完成 ?5. 監(jiān)控告警規(guī)則與面板配置# 1. 配置監(jiān)控告警規(guī)則 # 檢查現(xiàn)有告警規(guī)則 # kubectl get cm -n monitoring prometheus-rules -o yaml 2/dev/null | head -100 ? # 2. 配置 Alertmanager 告警路由 # 檢查 Alertmanager 配置 # kubectl get cm -n monitoring alertmanager-config -o yaml 2/dev/null ? # 3. 配置監(jiān)控告警渠道 # 企業(yè)微信/釘釘/郵件/短信告警渠道配置 # 檢查 Secret 中的告警渠道配置 # kubectl get secret -n monitoring alertmanager-secret -o yaml 2/dev/null ? # 4. 配置監(jiān)控面板 # 檢查 Grafana 數(shù)據(jù)源和面板 # kubectl get cm -n monitoring grafana-dashboards -o yaml 2/dev/null | head -50 ? # 5. 配置監(jiān)控?cái)?shù)據(jù)持久化 # 檢查 PVC 綁定狀態(tài) kubectl get pvc -n monitoring -o wide 2/dev/null # 預(yù)期PVC 已 Bound數(shù)據(jù)持久化正常 ? # 6. 配置監(jiān)控組件資源限制 # 檢查 Deployment/StatefulSet 資源限制 kubectl get deployment -n monitoring -o jsonpath{.items[*].spec.template.spec.containers[*].resources} 2/dev/null kubectl get statefulset -n monitoring -o jsonpath{.items[*].spec.template.spec.containers[*].resources} 2/dev/null ? # 7. 配置監(jiān)控組件高可用 # 檢查副本數(shù)和反親和性 kubectl get deployment -n monitoring -o jsonpath{.items[*].spec.replicas} 2/dev/null kubectl get statefulset -n monitoring -o jsonpath{.items[*].spec.replicas} 2/dev/null ? # 8. 配置監(jiān)控組件日志輪轉(zhuǎn) # 檢查日志配置 kubectl get cm -n monitoring -o name 2/dev/null | head -20 ?6. 驗(yàn)證操作結(jié)果與生成報(bào)告# 1. 驗(yàn)證監(jiān)控組件狀態(tài) kubectl get pods -n monitoring -o wide 2/dev/null # 預(yù)期所有監(jiān)控組件 Pod RunningREADY 正常 ? # 2. 驗(yàn)證監(jiān)控?cái)?shù)據(jù)采集 # 檢查 Prometheus Targets # kubectl port-forward -n monitoring svc/prometheus 9090:9090 # sleep 5 # curl -s http://localhost:9090/api/v1/targets | jq .data.activeTargets | length # 預(yù)期Targets 數(shù)量符合預(yù)期 ? # 3. 驗(yàn)證告警規(guī)則 # curl -s http://localhost:9090/api/v1/rules | jq .data.groups | length # 預(yù)期告警規(guī)則組已加載 ? # 4. 驗(yàn)證 Grafana 面板 # curl -s http://admin:adminlocalhost:3000/api/search | jq .[].title | head -20 # 預(yù)期監(jiān)控面板已配置 ? # 5. 驗(yàn)證 Alertmanager 告警 # curl -s http://localhost:9093/api/v2/alerts | jq length # 預(yù)期告警狀態(tài)正常 ? # 6. 驗(yàn)證監(jiān)控?cái)?shù)據(jù)持久化 kubectl get pvc -n monitoring -o wide 2/dev/null # 預(yù)期PVC Bound數(shù)據(jù)持久化正常 ? # 7. 驗(yàn)證監(jiān)控組件資源使用 kubectl top pods -n monitoring 2/dev/null # 預(yù)期資源使用在合理范圍內(nèi) ? # 8. 生成操作報(bào)告 echo 監(jiān)控操作報(bào)告 /tmp/monitor_operation_report.txt echo 操作時(shí)間: $(date) /tmp/monitor_operation_report.txt echo 監(jiān)控組件 Pod 數(shù): $(kubectl get pods -n monitoring --no-headers 2/dev/null | wc -l) /tmp/monitor_operation_report.txt echo 異常 Pod: $(kubectl get pods -n monitoring --no-headers 2/dev/null | grep -v Running | grep -v NAME | wc -l) /tmp/monitor_operation_report.txt echo PVC 數(shù): $(kubectl get pvc -n monitoring --no-headers 2/dev/null | wc -l) /tmp/monitor_operation_report.txt echo Service 數(shù): $(kubectl get svc -n monitoring --no-headers 2/dev/null | wc -l) /tmp/monitor_operation_report.txt echo ConfigMap 數(shù): $(kubectl get cm -n monitoring --no-headers 2/dev/null | wc -l) /tmp/monitor_operation_report.txt cat /tmp/monitor_operation_report.txt ? # 9. 清理 port-forward # pkill -f port-forward 2/dev/null ?驗(yàn)證流程# 1. 監(jiān)控組件狀態(tài)驗(yàn)證 kubectl get pods -n monitoring -o wide 2/dev/null # 預(yù)期所有監(jiān)控組件 Pod RunningREADY 正常 ? # 2. Prometheus 數(shù)據(jù)采集驗(yàn)證 # kubectl port-forward -n monitoring svc/prometheus 9090:9090 # sleep 5 # curl -s http://localhost:9090/api/v1/targets | jq .data.activeTargets[] | select(.health!up) | {job, health} # 預(yù)期所有 Targets 狀態(tài) up ? # 3. Prometheus 查詢(xún)驗(yàn)證 # curl -s http://localhost:9090/api/v1/query?queryup | jq .data.result | length # 預(yù)期查詢(xún)返回正常數(shù)據(jù) ? # 4. Grafana 訪問(wèn)驗(yàn)證 # curl -s http://admin:adminlocalhost:3000/api/health # 預(yù)期Grafana 健康 ? # 5. Alertmanager 狀態(tài)驗(yàn)證 # curl -s http://localhost:9093/api/v2/status | jq .cluster.status # 預(yù)期Alertmanager 集群正常 ? # 6. 監(jiān)控?cái)?shù)據(jù)持久化驗(yàn)證 kubectl get pvc -n monitoring -o wide 2/dev/null # 預(yù)期PVC Bound ? # 7. 監(jiān)控組件資源驗(yàn)證 kubectl top pods -n monitoring 2/dev/null # 預(yù)期資源使用合理 ? # 8. 操作報(bào)告驗(yàn)證 cat /tmp/monitor_operation_report.txt # 預(yù)期報(bào)告包含操作前后對(duì)比 ?排錯(cuò)方案Prometheus 啟動(dòng)失敗檢查配置文件語(yǔ)法、PVC 掛載、權(quán)限、端口沖突、鏡像版本、資源限制Prometheus 采集失敗 Target Down檢查目標(biāo)服務(wù)狀態(tài)、網(wǎng)絡(luò)連通性、防火墻端口、RBAC 權(quán)限、服務(wù)發(fā)現(xiàn)配置Prometheus 內(nèi)存溢出 OOM檢查指標(biāo)基數(shù)、采集頻率、保留時(shí)間、內(nèi)存限制、TSDB 配置、查詢(xún)優(yōu)化Prometheus CPU 占用過(guò)高檢查高基數(shù)指標(biāo)、復(fù)雜查詢(xún)、采集任務(wù)數(shù)量、規(guī)則評(píng)估頻率、資源限制Prometheus 磁盤(pán) IO 打滿(mǎn)檢查 TSDB 寫(xiě)入、壓縮、保留策略、磁盤(pán)性能、PVC 容量、IO 調(diào)度器Prometheus 配置熱更新失敗檢查配置文件語(yǔ)法、reload 接口、配置校驗(yàn)、ConfigMap 更新、Pod 重啟策略Prometheus 高可用數(shù)據(jù)不一致檢查遠(yuǎn)程讀寫(xiě)、聯(lián)邦集群、數(shù)據(jù)同步、時(shí)間漂移、存儲(chǔ)配置Prometheus 服務(wù)發(fā)現(xiàn)失敗檢查 RBAC 權(quán)限、API Server 連通性、服務(wù)發(fā)現(xiàn)配置、標(biāo)簽選擇器、網(wǎng)絡(luò)策略