2026-05-05 20:20:16 +08:00
|
|
|
|
---
|
|
|
|
|
|
tags: [microservice, distributed-tracing, opentelemetry, jaeger, skywalking]
|
|
|
|
|
|
create time: 2026-05-05
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
# 分布式链路追踪
|
|
|
|
|
|
|
|
|
|
|
|
## 概述
|
|
|
|
|
|
|
|
|
|
|
|
单个服务的日志只能告诉你局部信息。分布式链路追踪将一次请求跨越多个服务的完整调用链串起来，形成全局视图。
|
|
|
|
|
|
|
|
|
|
|
|
> [!question] 定位问题的困难
|
|
|
|
|
|
> 用户反映下单慢，你的系统由订单、支付、库存、会员 4 个服务串联而成。没有工具的情况下，你要怎么知道是哪个服务拖慢了整体响应时间？
|
|
|
|
|
|
|
|
|
|
|
|
## 核心概念
|
|
|
|
|
|
|
|
|
|
|
|
```mermaid
|
|
|
|
|
|
flowchart TB
|
|
|
|
|
|
Req["请求 trace_id=abc123"] --> Span1["Span #1<br/>API Gateway<br/>5ms"]
|
|
|
|
|
|
Span1 --> Span2["Span #2<br/>Order Service<br/>70ms"]
|
|
|
|
|
|
Span1 --> Span3["Span #3<br/>User Service<br/>15ms"]
|
|
|
|
|
|
Span2 --> Span4["Span #4<br/>Inventory DB Query<br/>60ms"]
|
|
|
|
|
|
|
|
|
|
|
|
style Span2 fill:#ff9999
|
|
|
|
|
|
style Span4 fill:#ffcc99
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
| 概念 | 说明 |
|
|
|
|
|
|
|------|------|
|
|
|
|
|
|
| **Trace** | 一次完整请求的调用链，由一个唯一的 `trace_id` 标识 |
|
|
|
|
|
|
| **Span** | 链路中的一个执行片段（如一次 HTTP 调用、一次 SQL 查询），有独立的 `span_id` |
|
|
|
|
|
|
| **Parent-Child** | Span 之间通过 `parent_span_id` 建立父子关系，形成树状结构 |
|
|
|
|
|
|
| **Context Propagation** | 通过 Header 传递 `trace_id`/`span_id`，贯穿整条链路 |
|
|
|
|
|
|
| **Sampling** | 不是每条请求都采样，按比例或策略选择，降低存储开销 |
|
|
|
|
|
|
|
|
|
|
|
|
### Trace/Span 数据结构
|
|
|
|
|
|
|
|
|
|
|
|
```json
|
|
|
|
|
|
{
|
|
|
|
|
|
"traceId": "abc123def456...",
|
|
|
|
|
|
"spans": [
|
|
|
|
|
|
{
|
|
|
|
|
|
"spanId": "span-001",
|
|
|
|
|
|
"parentId": null,
|
|
|
|
|
|
"operationName": "POST /orders",
|
|
|
|
|
|
"startTime": "2026-05-05T10:00:00.000Z",
|
|
|
|
|
|
"durationMs": 120,
|
|
|
|
|
|
"tags": {
|
|
|
|
|
|
"http.method": "POST",
|
|
|
|
|
|
"http.url": "/api/orders",
|
|
|
|
|
|
"http.status_code": 200
|
|
|
|
|
|
},
|
|
|
|
|
|
"logs": [
|
|
|
|
|
|
{
|
|
|
|
|
|
"timestamp": "2026-05-05T10:00:00.050Z",
|
|
|
|
|
|
"fields": [{"key": "event", "value": "order.created"}]
|
|
|
|
|
|
}
|
|
|
|
|
|
]
|
|
|
|
|
|
},
|
|
|
|
|
|
{
|
|
|
|
|
|
"spanId": "span-002",
|
|
|
|
|
|
"parentId": "span-001",
|
|
|
|
|
|
"operationName": "GetUserById (gRPC)",
|
|
|
|
|
|
"startTime": "2026-05-05T10:00:00.010Z",
|
|
|
|
|
|
"durationMs": 15,
|
|
|
|
|
|
"tags": {
|
|
|
|
|
|
"rpc.system": "grpc",
|
|
|
|
|
|
"rpc.service": "UserService",
|
|
|
|
|
|
"rpc.method": "GetUser"
|
|
|
|
|
|
}
|
|
|
|
|
|
}
|
|
|
|
|
|
]
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 上下文传播 (Context Propagation)
|
|
|
|
|
|
|
|
|
|
|
|
`trace_id` 如何从上游服务传递到下游？
|
|
|
|
|
|
|
|
|
|
|
|
### HTTP 场景：W3C Trace Context 标准
|
|
|
|
|
|
|
|
|
|
|
|
```json
|
|
|
|
|
|
// HTTP Header 中实际传递的字段
|
|
|
|
|
|
{
|
|
|
|
|
|
"traceparent": "00-abc123def456...-789ghi012jkl-01",
|
|
|
|
|
|
"tracestate": "congo=t61rcWkgMzE"
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
格式：`version-trace_id-span_id-flags`
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
// OpenTelemetry SDK 自动处理 context 注入和提取
|
|
|
|
|
|
func Middleware(next http.Handler) http.Handler {
|
|
|
|
|
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
|
|
|
|
// 从入站请求提取 trace context
|
|
|
|
|
|
ctx := propagator.Extract(r.Context(), headerReader{r.Header})
|
|
|
|
|
|
|
|
|
|
|
|
// 创建新的 span
|
|
|
|
|
|
ctx, span := tracer.Start(ctx, "handleOrder")
|
|
|
|
|
|
defer span.End()
|
|
|
|
|
|
|
|
|
|
|
|
// 将 trace context 注入到出站请求
|
|
|
|
|
|
r = r.WithContext(ctx)
|
|
|
|
|
|
propagator.Inject(ctx, headerWriter{r.Header})
|
|
|
|
|
|
|
|
|
|
|
|
next.ServeHTTP(w, r)
|
|
|
|
|
|
})
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### MQ 场景：Message Header 透传
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
// RocketMQ 生产者 - 自动注入 trace context
|
|
|
|
|
|
msg := rocketmq.NewMessage("order-created", payload)
|
|
|
|
|
|
msg.WithProperty("trace_id", currentTraceID) // 手动注入
|
|
|
|
|
|
msg.WithProperty("span_id", currentSpanID)
|
|
|
|
|
|
|
|
|
|
|
|
// RocketMQ 消费者 - 恢复 trace context
|
|
|
|
|
|
traceID := msg.GetProperty("trace_id")
|
|
|
|
|
|
span, _ := tracer.Start(ctx, "processOrderCreated",
|
|
|
|
|
|
trace.WithSpanKind(trace.SpanKindConsumer))
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 主流方案对比
|
|
|
|
|
|
|
|
|
|
|
|
| 方案 | 协议 | 存储后端 | 侵入程度 | 特色能力 |
|
|
|
|
|
|
|------|------|---------|---------|---------|
|
|
|
|
|
|
| **OpenTelemetry** | OTLP | Prometheus/Jaeger/Zipkin | SDK + Auto-Instrumentation | CNCF 标准，厂商中立，未来趋势 |
|
|
|
|
|
|
| **Jaeger** | Jaeger native | Cassandra/Elasticsearch | Agent / SDK | Uber 开源，UI 友好，支持业务 Tags |
|
|
|
|
|
|
| **SkyWalking** | SkyWalking | MySQL/Elasticsearch/ES | 零侵入 Java Agent | 国产，中文文档完善，APM 一体 |
|
|
|
|
|
|
|
|
|
|
|
|
> [!tip] 选型建议
|
|
|
|
|
|
> 新项目优先选 **OpenTelemetry**——它是行业标准，未来会被所有工具支持。如果团队需要开箱即用的 APM，**SkyWalking** 的 Java Agent 零侵入方案是快速上手的最佳选择。
|
|
|
|
|
|
|
|
|
|
|
|
## OpenTelemetry Go 实战
|
|
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
|
// 初始化 TracerProvider
|
|
|
|
|
|
provider := sdktrace.NewTracerProvider(
|
|
|
|
|
|
sdktrace.WithBatcherExporter(exporter), // 异步批量上报，不阻塞
|
|
|
|
|
|
)
|
|
|
|
|
|
defer provider.Shutdown(context.Background())
|
|
|
|
|
|
|
|
|
|
|
|
tracer := provider.Tracer("order-service")
|
|
|
|
|
|
|
|
|
|
|
|
func handleOrder(w http.ResponseWriter, r *http.Request) {
|
|
|
|
|
|
ctx, span := tracer.Start(r.Context(), "handleOrder")
|
|
|
|
|
|
defer span.End()
|
|
|
|
|
|
|
|
|
|
|
|
// 设置丰富的属性，方便查询和过滤
|
|
|
|
|
|
span.SetAttributes(
|
|
|
|
|
|
attribute.String("http.method", r.Method),
|
|
|
|
|
|
attribute.Int("http.status_code", http.StatusOK),
|
|
|
|
|
|
attribute.String("user.id", getUserID(r)),
|
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
// 调用下游服务（trace context 自动传播）
|
|
|
|
|
|
resp, err := callInventoryService(ctx)
|
|
|
|
|
|
if err != nil {
|
|
|
|
|
|
span.RecordError(err)
|
|
|
|
|
|
span.SetStatus(codes.Error, err.Error())
|
|
|
|
|
|
w.WriteHeader(http.StatusBadGateway)
|
|
|
|
|
|
return
|
|
|
|
|
|
}
|
|
|
|
|
|
w.Write(resp.Body)
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 采样策略
|
|
|
|
|
|
|
|
|
|
|
|
```mermaid
|
|
|
|
|
|
flowchart LR
|
|
|
|
|
|
AllReq["全部请求"] --> Sampler{"采样决策"}
|
|
|
|
|
|
Sampler -->|"100%"| Core["核心链路全量采集"]
|
|
|
|
|
|
Sampler -->|"5~10%"| Normal["普通路径随机采样"]
|
|
|
|
|
|
Sampler -->|"0%"| Health["健康检查/内部心跳"]
|
|
|
|
|
|
|
|
|
|
|
|
Core --> Storage["Trace 存储"]
|
|
|
|
|
|
Normal --> Storage
|
|
|
|
|
|
Health --> Drop["丢弃"]
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
| 环境 | 采样率 | 理由 |
|
|
|
|
|
|
|------|--------|------|
|
|
|
|
|
|
| **开发 / 测试** | 100% | 方便调试，无存储压力 |
|
|
|
|
|
|
| **生产 - 核心链路** | 100% | 下单、支付等关键路径必须全量 |
|
|
|
|
|
|
| **生产 - 普通路径** | 5~10% | 平衡成本和覆盖率 |
|
|
|
|
|
|
| **生产 - 错误链路** | 100% | 出错时的请求优先保留 |
|
|
|
|
|
|
|
|
|
|
|
|
> [!note] 基于错误的智能采样
|
|
|
|
|
|
>
|
|
|
|
|
|
> 高级做法：**正常路径低采样，一旦检测到错误立即提升当前请求的采样率**。这样既省了存储，又能在出问题时有足够的数据回溯。
|
|
|
|
|
|
|
|
|
|
|
|
## 渐进式落地路线
|
|
|
|
|
|
|
|
|
|
|
|
> [!tip] 不要试图一开始就采集全部 Span
|
|
|
|
|
|
>
|
|
|
|
|
|
> 1. **第一步**：先接 Tracing，覆盖核心链路（下单、支付），rate=100%
|
|
|
|
|
|
> 2. **第二步**：加入 Metrics 监控（Prometheus + Grafana）
|
|
|
|
|
|
> 3. **第三步**：集中 Logging（Loki / ELK），与 trace_id 关联
|
|
|
|
|
|
> 4. **第四步**：全量上 OpenTelemetry Collector，统一管理
|
|
|
|
|
|
|
|
|
|
|
|
## 关联笔记
|
|
|
|
|
|
|
2026-05-15 16:26:14 +08:00
|
|
|
|
- [[02-服务治理/01-API网关]] — API Gateway 可以在入口处注入 trace_id
|
|
|
|
|
|
- [[04-可观测性/03-链路追踪]] — 更详细的链路追踪设计方法论
|
|
|
|
|
|
- [[02-服务治理/07-配置管理]] — 配置中心的动态刷新可以联动调整采样率
|
