398 lines
13 KiB
Markdown
398 lines
13 KiB
Markdown
|
---
|
|||
|
|
tags: [microservice, kubernetes, cicd, container, sre, observability]
|
|||
|
|
create time: 2026-04-29 12:05
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 部署运维
|
|||
|
|
|
|||
|
|
## 概述
|
|||
|
|
|
|||
|
|
微服务架构下，服务数量从几个增长到几百个。**人工运维完全不可行**。本文涵盖容器化编排、K8s 资源管理、发布策略、弹性伸缩、CI/CD 流水线、日志告警和 SRE 核心概念。
|
|||
|
|
|
|||
|
|
## 容器化与编排
|
|||
|
|
|
|||
|
|
### Docker 容器 — 标准交付单元
|
|||
|
|
|
|||
|
|
```dockerfile
|
|||
|
|
# Go 多阶段构建示例
|
|||
|
|
FROM golang:1.22 AS builder
|
|||
|
|
WORKDIR /app
|
|||
|
|
COPY . .
|
|||
|
|
RUN CGO_ENABLED=0 GOOS=linux go build -o server .
|
|||
|
|
|
|||
|
|
FROM alpine:latest
|
|||
|
|
COPY --from=builder /app/server /server
|
|||
|
|
EXPOSE 8080
|
|||
|
|
CMD ["/server"]
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
> [!tip] 关键实践
|
|||
|
|
> - **多阶段构建**：镜像只包含运行时产物，体积极大缩小
|
|||
|
|
> - **非 root 用户运行**：安全最佳实践
|
|||
|
|
> - `.dockerignore`：排除不必要的文件（git、vendor、测试文件）
|
|||
|
|
|
|||
|
|
### Kubernetes 核心概念
|
|||
|
|
|
|||
|
|
```mermaid
|
|||
|
|
graph TB
|
|||
|
|
subgraph CLUSTER["K8s Cluster"]
|
|||
|
|
master["Master Node<br/>API Server / Scheduler / ETCD"]
|
|||
|
|
|
|||
|
|
subgraph NODES["Worker Nodes"]
|
|||
|
|
N1[Node A]
|
|||
|
|
N2[Node B]
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
master --> N1
|
|||
|
|
master --> N2
|
|||
|
|
|
|||
|
|
subgraph PODS1["Deployment: order-service"]
|
|||
|
|
P1[Pod 1]
|
|||
|
|
P2[Pod 2]
|
|||
|
|
P3[Pod 3]
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
subgraph PODS2["Deployment: payment-service"]
|
|||
|
|
Q1[Pod 1]
|
|||
|
|
Q2[Pod 2]
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
N1 --> P1 & Q1
|
|||
|
|
N2 --> P2 & P3 & Q2
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
Svc["Service: order-service<br/>Load Balancer → Pods"]
|
|||
|
|
Svc --> P1 & P2 & P3
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
| K8s 对象 | 作用 |
|
|||
|
|
|---------|------|
|
|||
|
|
| Pod | 最小部署单元，一个或多个容器 |
|
|||
|
|
| Deployment | 管理 Pod 的声明式更新和扩缩容 |
|
|||
|
|
| Service | 稳定的网络入口，实现负载均衡 |
|
|||
|
|
| Ingress | HTTP/HTTPS 路由规则（外部访问入口） |
|
|||
|
|
| ConfigMap / Secret | 配置注入，与代码分离 |
|
|||
|
|
| HPA (Horizontal Pod Autoscaler) | 根据 CPU/自定义指标自动扩缩 |
|
|||
|
|
|
|||
|
|
### 资源管理与健康检查
|
|||
|
|
|
|||
|
|
K8s 调度 Pod 的核心依据是 **资源请求 (Requests)**，而决定 pod 生死的是 **探针 (Probes)**。这两个概念配合使用才能保障服务稳定运行。
|
|||
|
|
|
|||
|
|
```yaml
|
|||
|
|
# Deployment 核心片段：资源配置 + 探针
|
|||
|
|
spec:
|
|||
|
|
replicas: 3
|
|||
|
|
template:
|
|||
|
|
spec:
|
|||
|
|
containers:
|
|||
|
|
- name: order-service
|
|||
|
|
image: registry.example.com/order:v1.2.3
|
|||
|
|
resources:
|
|||
|
|
requests: # 调度依据：K8s 保证至少有这些资源
|
|||
|
|
cpu: "250m" # 0.25 核
|
|||
|
|
memory: "256Mi" # 256 MB
|
|||
|
|
limits: # 硬上限：超过则 OOMKill / CPU Throttle
|
|||
|
|
cpu: "500m"
|
|||
|
|
memory: "512Mi"
|
|||
|
|
livenessProbe: # 活体检测：死了就重启
|
|||
|
|
httpGet:
|
|||
|
|
path: /healthz
|
|||
|
|
port: 8080
|
|||
|
|
initialDelaySeconds: 15
|
|||
|
|
periodSeconds: 10
|
|||
|
|
readinessProbe: # 就绪检测：未就绪就不进负载均衡
|
|||
|
|
httpGet:
|
|||
|
|
path: /ready
|
|||
|
|
port: 8080
|
|||
|
|
initialDelaySeconds: 5
|
|||
|
|
periodSeconds: 5
|
|||
|
|
startupProbe: # 启动检测：慢启动服务友好
|
|||
|
|
httpGet:
|
|||
|
|
path: /healthz
|
|||
|
|
port: 8080
|
|||
|
|
failureThreshold: 30
|
|||
|
|
periodSeconds: 10
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
> [!tip] Probe 选择指南
|
|||
|
|
>
|
|||
|
|
> | 探针类型 | 触发条件 | 后果 | 适用场景 |
|
|||
|
|
> |---------|---------|------|---------|
|
|||
|
|
> | **Liveness** | `/healthz` 返回非 2xx | K8s **重启容器** | 死锁、无法恢复的崩溃 |
|
|||
|
|
> | **Readiness** | `/ready` 返回非 2xx | **摘除 Service 流量** | 依赖 DB 连接池未就绪、热加载进行中 |
|
|||
|
|
> | **Startup** | 首次成功前一直失败 | **不重启，只等待** | 大模型初始化、JVM cold start 等慢启动场景 |
|
|||
|
|
|
|||
|
|
> [!question] 思考
|
|||
|
|
> 如果一个服务的 `/healthz` 因为数据库连接超时而持续返回 503，K8s 会怎么做？
|
|||
|
|
|
|||
|
|
**答案**：Liveness Probe 会认为容器挂了并反复重启它 — 这就是经典的 **CrashLoopBackOff**。正确做法是让 `/healthz` 做降级判断（DB 不可用时返回 200 + 标注降级），用 `/ready` 来摘除流量。Liveness 应该只对"进程已死"的情况敏感，对"性能下降"保持宽容。
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 发布策略
|
|||
|
|
|
|||
|
|
微服务需要独立发布，如何在不停服的情况下完成更新？
|
|||
|
|
|
|||
|
|
### 蓝绿发布
|
|||
|
|
|
|||
|
|
```mermaid
|
|||
|
|
stateDiagram-v2
|
|||
|
|
[*] --> Blue: 初始状态
|
|||
|
|
Blue --> DeployGreen: 部署新版本到 Green
|
|||
|
|
DeployGreen --> TestGreen: 灰度验证
|
|||
|
|
TestGreen --> Switch: 切换流量
|
|||
|
|
Switch --> Green: 全部流量到新版
|
|||
|
|
Green --> CleanupGreen: 删除旧版 Blue
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- 优点：**回滚极速**（切回 Blue 即可），验证充分
|
|||
|
|
- 缺点：资源翻倍，需要两套环境
|
|||
|
|
|
|||
|
|
### 金丝雀发布 (Canary)
|
|||
|
|
|
|||
|
|
```mermaid
|
|||
|
|
graph LR
|
|||
|
|
A["95% traffic to stable"]
|
|||
|
|
B["5% traffic to canary"]
|
|||
|
|
C["Metrics normal?"]
|
|||
|
|
D["Gradually expand to 20% → 50% → 100%"]
|
|||
|
|
E["Metrics abnormal → Auto rollback"]
|
|||
|
|
|
|||
|
|
A --> B
|
|||
|
|
B --> C
|
|||
|
|
C -- "yes" --> D
|
|||
|
|
C -- "no" --> E
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- 优点：资源效率高，风险渐进暴露
|
|||
|
|
- 缺点：流程复杂，需要完善的监控配合
|
|||
|
|
|
|||
|
|
### 两种策略选择
|
|||
|
|
|
|||
|
|
> [!question] 思考
|
|||
|
|
> 你的团队应该用蓝绿还是金丝雀？
|
|||
|
|
|
|||
|
|
**答案线索**：看你们的**监控成熟度和发布频率**。低频次（每周/每月）、监控完善时用金丝雀；高频次（每天多次）或想简化流程时用蓝绿。
|
|||
|
|
|
|||
|
|
## 弹性伸缩
|
|||
|
|
|
|||
|
|
```mermaid
|
|||
|
|
graph TD
|
|||
|
|
Metrics["CPU / Memory / Custom QPS"] --> Trigger{Threshold met?}
|
|||
|
|
Trigger -- "yes" --> ScaleUp["Scale up new Pod"]
|
|||
|
|
Trigger -- "no" --> Keep["Keep current replicas"]
|
|||
|
|
ScaleDown{"Load decreased?"} -- "yes" --> ScaleDownPod["Scale down Pod"]
|
|||
|
|
ScaleDown -- "no" --> Keep
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**注意事项**：
|
|||
|
|
- **预热时间**：新 Pod 启动后不能立刻算入统计，否则可能反复扩缩
|
|||
|
|
- **优雅关闭**：HPA 缩容前，Pod 需要先摘除 Service 流量再退出
|
|||
|
|
- **预留容量**：不要将集群利用率拉到 100%，给突发流量留缓冲
|
|||
|
|
|
|||
|
|
## CI/CD 流水线
|
|||
|
|
|
|||
|
|
```mermaid
|
|||
|
|
graph LR
|
|||
|
|
CODE["代码提交"]
|
|||
|
|
TEST["单元测试 + 集成测试"]
|
|||
|
|
SCAN["安全扫描 + 代码质量"]
|
|||
|
|
BUILD["构建 Docker 镜像"]
|
|||
|
|
PUSH["推送镜像仓库"]
|
|||
|
|
DEPLOY["CI→Staging → 审批 → Production"]
|
|||
|
|
|
|||
|
|
CODE --> TEST --> SCAN --> BUILD --> PUSH --> DEPLOY
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
> [!summary] CI/CD 设计原则
|
|||
|
|
>
|
|||
|
|
> - **一次构建，多处部署**：镜像不随环境重新编译，只改 K8s ConfigMap/环境变量
|
|||
|
|
> - **语义化版本号**：镜像 tag 用 `v1.2.3`，tag 即版本溯源
|
|||
|
|
> - **自动化测试覆盖率要求**：合并 PR 前必须通过，否则不允许发布
|
|||
|
|
|
|||
|
|
### Pipeline 实战：GitHub Actions
|
|||
|
|
|
|||
|
|
```yaml
|
|||
|
|
# .github/workflows/deploy.yml
|
|||
|
|
name: Deploy order-service
|
|||
|
|
on:
|
|||
|
|
push:
|
|||
|
|
branches: [main]
|
|||
|
|
paths:
|
|||
|
|
- "services/order/**" # 仅该目录变更才触发
|
|||
|
|
|
|||
|
|
jobs:
|
|||
|
|
build-and-push:
|
|||
|
|
runs-on: ubuntu-latest
|
|||
|
|
steps:
|
|||
|
|
- uses: actions/checkout@v4
|
|||
|
|
- name: Build & push image
|
|||
|
|
run: |
|
|||
|
|
docker build -t registry/order:${{ github.sha }} .
|
|||
|
|
docker tag registry/order:${{ github.sha }} \
|
|||
|
|
registry/order:v$(git describe --tags --abbrev=0)
|
|||
|
|
docker push registry/order:${{ github.sha }}
|
|||
|
|
|
|||
|
|
deploy-staging:
|
|||
|
|
needs: build-and-push
|
|||
|
|
runs-on: ubuntu-latest
|
|||
|
|
environment: staging
|
|||
|
|
steps:
|
|||
|
|
- name: Apply K8s manifests
|
|||
|
|
run: |
|
|||
|
|
kubectl set image deployment/order-service \
|
|||
|
|
order=registry/order:${{ github.sha }}
|
|||
|
|
kubectl rollout status deployment/order-service --timeout=120s
|
|||
|
|
|
|||
|
|
deploy-production:
|
|||
|
|
needs: deploy-staging
|
|||
|
|
runs-on: ubuntu-latest
|
|||
|
|
environment: production
|
|||
|
|
steps:
|
|||
|
|
- name: Canary release (10% → 50% → 100%)
|
|||
|
|
run: |
|
|||
|
|
kubectl patch canary order-service --type merge \
|
|||
|
|
-p '{"spec":{"weight":10}}'
|
|||
|
|
# ... 等待监控确认，逐步放大流量
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
> [!tip] Pipeline 设计要点
|
|||
|
|
>
|
|||
|
|
> - **路径过滤**：只对相关服务的代码变更触发构建，避免全量重建
|
|||
|
|
> - **Commit SHA 作为镜像 tag**：保证精确回滚 — `git revert` 之后用同一 SHA 拉取旧镜像
|
|||
|
|
> - **分阶段部署**：Staging → Production 的审批关卡是最后的防线，不要跳过
|
|||
|
|
|
|||
|
|
## 日志管理与告警
|
|||
|
|
|
|||
|
|
微服务单实例崩溃不可怕 — 可怕的是 **不知道哪一台、为什么挂了**。日志和告警是运维的眼睛。
|
|||
|
|
|
|||
|
|
### 日志架构选型
|
|||
|
|
|
|||
|
|
```mermaid
|
|||
|
|
graph LR
|
|||
|
|
App["业务应用"] -->|stdout/stderr| K8sPod[Pod 容器日志]
|
|||
|
|
K8sPod --> Filebeat[采集器: Filebeat / FluentBit]
|
|||
|
|
Filebeat --> ES["Elasticsearch / Loki"]
|
|||
|
|
ES --> Grafana["Grafana 查询面板"]
|
|||
|
|
|
|||
|
|
App -->|structured log| sidecar[Sidecar 辅助日志]
|
|||
|
|
sidecar --> Filebeat
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**业界两种主流方案对比：**
|
|||
|
|
|
|||
|
|
| 维度 | ELK (Elasticsearch) | EFK/Loki |
|
|||
|
|
|------|---------------------|----------|
|
|||
|
|
| 存储成本 | 高（全文索引） | 低（索引 label + 对象存储原文） |
|
|||
|
|
| 查询性能 | 毫秒级全文检索 | 按 label 过滤较快，复杂查询慢 |
|
|||
|
|
| 运维复杂度 | 高（ES 集群维护） | 低（Loki 无索引） |
|
|||
|
|
| 适用规模 | 百万条/天以上 | 万 ~ 百万条/天 |
|
|||
|
|
|
|||
|
|
> [!important] 结构化日志
|
|||
|
|
>
|
|||
|
|
> 无论选择哪家方案，**日志格式必须是结构化的**（JSON），否则后期处理全是手工活：
|
|||
|
|
> ```json
|
|||
|
|
> {
|
|||
|
|
> "timestamp": "2026-04-29T10:30:00Z",
|
|||
|
|
> "level": "error",
|
|||
|
|
> "service": "order-service",
|
|||
|
|
> "trace_id": "abc-123-def",
|
|||
|
|
> "msg": "payment gateway timeout",
|
|||
|
|
> "duration_ms": 5023
|
|||
|
|
> }
|
|||
|
|
> ```
|
|||
|
|
|
|||
|
|
### 告警策略设计
|
|||
|
|
|
|||
|
|
> [!question] 思考
|
|||
|
|
> 如果一个 Pod CPU 使用率从 20% 飙升到 90%，你应该立刻打电话叫值班工程师吗？
|
|||
|
|
|
|||
|
|
不一定。**好的告警应该是" actionable"的** — 能让人立刻采取行动，而不是单纯告诉你"出事了"。
|
|||
|
|
|
|||
|
|
```mermaid
|
|||
|
|
mindmap
|
|||
|
|
root((告警设计原则))
|
|||
|
|
区分等级
|
|||
|
|
P0: 立即响应 (页面电话)
|
|||
|
|
P1: 当天处理 (IM 消息)
|
|||
|
|
P2: 本周修复 (工单)
|
|||
|
|
抑制噪音
|
|||
|
|
相关告警聚合: 一个根因一条告警
|
|||
|
|
静默期: 重启后的自动恢复不打扰
|
|||
|
|
附带上下文
|
|||
|
|
告警信息包含: 什么服务 · 哪个指标 · 当前值 vs 阈值
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## SRE 核心概念
|
|||
|
|
|
|||
|
|
站点可靠性工程 (SRE) 把运维问题看作**软件工程问题**。它的核心理念是用 SLI/SLO/SLA 来量化服务质量，避免"我觉得系统很慢"这类模糊描述。
|
|||
|
|
|
|||
|
|
### SLI / SLO / SLA
|
|||
|
|
|
|||
|
|
| 术语 | 全称 | 定义 | 谁定义 |
|
|||
|
|
|------|------|------|--------|
|
|||
|
|
| **SLI** | Service Level Indicator | **实际度量**：用户请求的成功率是多少？ | 观测系统自动产出 |
|
|||
|
|
| **SLO** | Service Level Objective | **内部目标**：我们承诺达到 99.9% 可用性 | SRE + 研发制定 |
|
|||
|
|
| **SLA** | Service Level Agreement | **对外合同**：达不到就赔钱 | 法务 + 商务制定 |
|
|||
|
|
|
|||
|
|
> [!tip] 实用比例
|
|||
|
|
>
|
|||
|
|
> - **99.9% (三个九)** = 每年约 8.76 小时停机 — 适合大多数后端服务
|
|||
|
|
> - **99.95%** = 每年约 4.38 小时 — 核心交易链路
|
|||
|
|
> - **99.99% (四个九)** = 每年约 52 分钟 — 金融级系统
|
|||
|
|
> - **99%** 意味着每月近 7 小时不可用 — **几乎等于没有可用性目标**
|
|||
|
|
|
|||
|
|
### 错误预算 (Error Budget)
|
|||
|
|
|
|||
|
|
这是 SRE 最核心的机制 — **允许犯错，但设上限**：
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
错误预算 = 1 - SLO
|
|||
|
|
|
|||
|
|
SLO = 99.9% → 错误预算 = 0.1% → 每月允许 21 分钟停机
|
|||
|
|
SLO = 99.99% → 错误预算 = 0.01% → 每月仅 4 分钟停机
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**基于错误预算的决策逻辑：**
|
|||
|
|
|
|||
|
|
```mermaid
|
|||
|
|
flowchart TD
|
|||
|
|
Budget{"错误预算剩余 > 50%?"}
|
|||
|
|
|
|||
|
|
Budget -- "是" --> Aggressive["可激进发布: <br/>金丝雀 + 自动扩缩"]
|
|||
|
|
Budget -- "< 50%" --> Conservative["保守发布: <br/>蓝绿 + 全量人工审查"]
|
|||
|
|
Budget -- "< 10%" --> Freeze["冻结发布: <br/>优先修复稳定性<br/>不允许新功能上线"]
|
|||
|
|
|
|||
|
|
Aggressive --> Monitor["持续监控指标"]
|
|||
|
|
Conservative --> Monitor
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
> [!summary] SRE 心法
|
|||
|
|
>
|
|||
|
|
> 1. **用户视角定义 SLO**：不是"API P99 < 200ms"，而是"用户在 3G 网络下打开页面 < 2s"
|
|||
|
|
> 2. **错误预算用完 = 停止功能开发**：全力修 bug 加稳定性
|
|||
|
|
> 3. **Blameless Postmortem**：事故复盘不问"谁干的"，问"流程哪里可以改进"
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 运维 Checklist
|
|||
|
|
|
|||
|
|
每次上线前快速过一遍这个清单，避免低级事故：
|
|||
|
|
|
|||
|
|
| # | 检查项 | 说明 |
|
|||
|
|
|---|--------|------|
|
|||
|
|
| 1 | **探针已配置** | liveness/readiness/startup probe 都已设定 |
|
|||
|
|
| 2 | **资源 limits 已设** | 防止单个 Pod 拖垮整台机器 |
|
|||
|
|
| 3 | **日志为 JSON 格式** | 可被采集器解析 |
|
|||
|
|
| 4 | **追踪 ID 透传** | trace_id 在跨服务调用链中不丢失 |
|
|||
|
|
| 5 | **回滚预案明确** | 知道怎么回到上一版本，且演练过 |
|
|||
|
|
| 6 | **告警已配置** | 关键指标异常时有人收到通知 |
|
|||
|
|
| 7 | **数据迁移有回退脚本** | DB schema 变更要兼容新老版本共存 |
|
|||
|
|
|
|||
|
|
## 关联笔记
|
|||
|
|
|
|||
|
|
- [[02-服务治理]] — K8s Service 提供原生的服务发现和负载均衡
|
|||
|
|
- [[04-可观测性]] — K8s Liveness/Readiness Probe 是可观测性的基础
|