Prometheus监控体系搭建指南

Prometheus监控体系搭建指南

在微服务和容器化架构中，监控是保障系统稳定运行的关键。Prometheus是目前最流行的开源监控方案，配合Grafana可以实现强大的可视化监控。本文将介绍Prometheus监控体系的搭建方法。

一、Prometheus核心概念

# prometheus.yml - 基础配置
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "myapp"
    metrics_path: /metrics
    static_configs:
      - targets: ["localhost:8080"]
  
  - job_name: "node-exporter"
    static_configs:
      - targets: ["localhost:9100"]
  
  # 服务发现 - Consul
  - job_name: "consul-services"
    consul_sd_configs:
      - server: "localhost:8500"
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*monitoring.*
        action: keep

二、应用指标埋点

// Go应用埋点
import "github.com/prometheus/client_golang/prometheus"

var (
    httpRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total HTTP requests",
        },
        []string{"method", "path", "status"},
    )
    
    httpRequestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "http_request_duration_seconds",
            Help:    "HTTP request duration",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method", "path"},
    )
)

// 中间件记录指标
func metricsMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        next.ServeHTTP(w, r)
        duration := time.Since(start).Seconds()
        
        httpRequestsTotal.WithLabelValues(r.Method, r.URL.Path, status).Inc()
        httpRequestDuration.WithLabelValues(r.Method, r.URL.Path).Observe(duration)
    })
}

三、告警规则

# alert_rules.yml
groups:
  - name: app-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m]) 
          / rate(http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ labels.path }}"
          description: "Error rate is {{ value | humanizePercentage }}"
      
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High latency P95"

# Alertmanager配置
route:
  receiver: "default"
  group_by: ["alertname", "severity"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: "default"
    webhook_configs:
      - url: "http://alertmanager-webhook:8080/alert"

四、Grafana看板

// 常用PromQL查询

// QPS - 每秒请求数
rate(http_requests_total[5m])

// P95延迟
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

// 错误率
sum(rate(http_requests_total{status=~"5.."}[5m])) 
/ sum(rate(http_requests_total[5m]))

// CPU使用率
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

// 内存使用率
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

// 磁盘使用率
(1 - node_filesystem_avail_bytes{fstype!~"tmpfs"} / node_filesystem_size_bytes) * 100

完善的监控体系是系统稳定运行的保障。Prometheus+Grafana的组合功能强大且社区活跃，建议从基础指标开始，逐步完善告警规则和看板，建立起全方位的监控体系。