Init
This commit is contained in:
@@ -0,0 +1,160 @@
|
||||
---
|
||||
tags: [microservice, metrics, prometheus, grafana, sre]
|
||||
create time: 2026-05-05
|
||||
---
|
||||
|
||||
# Metrics 监控
|
||||
|
||||
## 概述
|
||||
|
||||
Metrics 回答的问题是：**系统现在健康吗？**——通过聚合后的数值揭示趋势。
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
App["应用服务"] -->|Push / Scrape| PROM[(Prometheus)]
|
||||
PROM --> GRAF["Grafana Dashboard"]
|
||||
PROM --> ALERT["Alertmanager"]
|
||||
|
||||
style PROM fill:#e3f2fd
|
||||
style GRAF fill:#fff3e0
|
||||
style ALERT fill:#fce4ec
|
||||
```
|
||||
|
||||
## 指标类型
|
||||
|
||||
| 类型 | 含义 | 特点 | 示例 |
|
||||
|------|------|------|------|
|
||||
| **Counter** | 只增不减的计数器 | 可 Reset（重启） | `http_requests_total` |
|
||||
| **Gauge** | 可升可降的仪表盘 | 反映当前状态 | `queue_depth`, `cpu_temp` |
|
||||
| **Histogram** | 样本分布，自动分桶 | 计算 P50/P90/P99 | `api_latency_seconds` |
|
||||
| **Summary** | 类似 Histogram，客户端算分位 | Go SDK 默认类型 | `grpc_duration_seconds` |
|
||||
|
||||
### Counter vs Gauge 场景选择
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Q{"这个值是<br/>只增不减的吗？"}
|
||||
|
||||
Q -- "是" --> C["Counter<br/>请求数、错误数、订单量"]
|
||||
Q -- "否" --> G["Gauge<br/>在线用户数、队列长度、内存使用量"]
|
||||
|
||||
style C fill:#c8e6c9
|
||||
style G fill:#bbdefb
|
||||
```
|
||||
|
||||
## RED 方法 (针对有状态服务)
|
||||
|
||||
适用于 API、微服务等有明确请求/响应的服务。
|
||||
|
||||
| 指标 | 公式 | 说明 |
|
||||
|------|------|------|
|
||||
| **Rate** | `rate(http_requests_total[5m])` | 每秒请求量 (QPS) |
|
||||
| **Errors** | `rate(http_requests_total{status="5xx"}[5m])` | 每秒错误数 |
|
||||
| **Duration** | `histogram_quantile(0.99, rate(api_latency_bucket[5m]))` | P99 响应时间 |
|
||||
|
||||
> [!tip] PromQL 关键函数
|
||||
>
|
||||
> - `rate()` — 计算 Counter 每秒增长率（必须用于 Counter）
|
||||
> - `irate()` — 即时速率，对突发更敏感
|
||||
> - `histogram_quantile(0.99, ...)` — 从直方图计算分位数
|
||||
> - `increase()` — 时间段内的增长总量
|
||||
> - `avg() / max() / min()` — 基础聚合函数
|
||||
|
||||
## USE 方法 (针对基础设施)
|
||||
|
||||
适用于 CPU、内存、网络、磁盘等底层资源监控。
|
||||
|
||||
| 指标 | 说明 | Grafana PromQL 示例 |
|
||||
|------|------|---------------------|
|
||||
| **Utilization** | 使用率 | `1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))` |
|
||||
| **Saturation** | 饱和度 | `node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes` |
|
||||
| **Errors** | 错误数 | `rate(node_network_receive_errs_total[5m])` |
|
||||
|
||||
## Go Prometheus 集成
|
||||
|
||||
```go
|
||||
var (
|
||||
httpRequestsTotal = prometheus.NewCounterVec(
|
||||
prometheus.CounterOpts{
|
||||
Name: "http_requests_total",
|
||||
Help: "Total HTTP requests by method and status",
|
||||
},
|
||||
[]string{"method", "status"},
|
||||
)
|
||||
|
||||
apiLatency = prometheus.NewHistogramVec(
|
||||
prometheus.HistogramOpts{
|
||||
Name: "api_latency_seconds",
|
||||
Help: "API latency distribution",
|
||||
Buckets: prometheus.DefBuckets, // [0.005, 0.01, ..., 10]
|
||||
},
|
||||
[]string{"endpoint"},
|
||||
)
|
||||
)
|
||||
|
||||
func init() {
|
||||
prometheus.MustRegister(httpRequestsTotal, apiLatency)
|
||||
}
|
||||
|
||||
// Middleware 中使用
|
||||
func MetricsMiddleware(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
start := time.Now()
|
||||
next.ServeHTTP(w, r)
|
||||
|
||||
duration := time.Since(start).Seconds()
|
||||
httpRequestsTotal.WithLabelValues(r.Method, fmt.Sprintf("%d", w.Status())).Inc()
|
||||
apiLatency.WithLabelValues(r.URL.Path).Observe(duration)
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## Dashboard 设计原则
|
||||
|
||||
一个优秀的 Dashboard 应该让任何团队成员在 **30 秒内**了解服务的整体状态：
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph DASH["Service Health Dashboard"]
|
||||
ROW1["🔴 可用性 & 错误率 — 第一眼判断"]
|
||||
ROW2["🟡 性能指标 — P50/P90/P99 趋势"]
|
||||
ROW3["🔵 基础设施 — CPU/内存/连接数"]
|
||||
ROW4["⚫ 业务指标 — 订单量/支付成功率"]
|
||||
end
|
||||
|
||||
ROW1 --> JUDGE{是否异常?}
|
||||
ROW2 --> JUDGE
|
||||
ROW3 --> JUDGE
|
||||
ROW4 --> JUDGE
|
||||
|
||||
JUDGE --"否" --> NORMAL["一切正常 ✓"]
|
||||
JUDGE --"是" --> ALERT["触发告警 → On-Call"]
|
||||
```
|
||||
|
||||
### Dashboard 布局模板
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ Row 1: 🔴 Service Availability │
|
||||
│ ├─ QPS (Rate) ┌─ Error Rate (%) │
|
||||
│ ├─ Active Connections └─ 5xx Count │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Row 2: 🟡 Performance │
|
||||
│ ├─ P50 Latency ┌─ P90 Latency │
|
||||
│ ├─ P99 Latency └─ Slow Requests (>1s) │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Row 3: 🔵 Infrastructure │
|
||||
│ ├─ CPU % ┌─ Memory Usage │
|
||||
│ ├─ GC Pause Time └─ Goroutine Count │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Row 4: ⚫ Business Metrics │
|
||||
│ ├─ Orders/Minute ┌─ Payment Success Rate │
|
||||
│ └─ New Users/Day └─ Failed Transactions │
|
||||
└─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 关联笔记
|
||||
|
||||
- [[04-可观测性/04-告警管理]] — Metrics 是告警的基础数据来源
|
||||
- [[04-可观测性/03-链路追踪]] — Tracing 与 Metrics 互补，定位具体故障
|
||||
- [[05-部署运维/04-SRE实践]] — SLO 基于 Metrics 数据
|
||||
Reference in New Issue
Block a user