Init
This commit is contained in:
@@ -0,0 +1,202 @@
|
||||
---
|
||||
tags: [kubernetes, pod, deployment, probe, container-orchestration]
|
||||
create time: 2026-05-18 00:30
|
||||
---
|
||||
|
||||
# K8s 核心概念与 Deployment — 教程
|
||||
|
||||
## 概述
|
||||
|
||||
本文档带你从零理解 Kubernetes 的核心对象模型,并掌握 Deployment 这个最常用的工作负载控制器。更多进阶主题(资源配置、服务发现、调度)分散在后续章节中。总览和决策矩阵见 [[../02-Kubernetes]]。
|
||||
|
||||
## K8s 架构全景
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph CONTROL["控制面 Control Plane"]
|
||||
APISERVER["API Server<br/>唯一入口"]
|
||||
SCHEDULER["Scheduler<br/>调度决策"]
|
||||
CMGR["Controller Manager<br/>状态协调"]
|
||||
ETCD["etcd<br/>分布式 KV 存储"]
|
||||
|
||||
APISERVER --> SCHEDULER
|
||||
APISERVER --> CMGR
|
||||
APISERVER --> ETCD
|
||||
CMGR --> ETCD
|
||||
end
|
||||
|
||||
subgraph WORKER["工作节点 Worker Node"]
|
||||
N1["Node A<br/>kubelet + kube-proxy"]
|
||||
N2["Node B<br/>kubelet + kube-proxy"]
|
||||
end
|
||||
|
||||
APISERVER -.->|监听变动| N1
|
||||
APISERVER -.->|监听变动| N2
|
||||
|
||||
style CONTROL fill:#e3f2fd
|
||||
style ETCD fill:#c8e6c9
|
||||
style WORKER fill:#fff3e0
|
||||
```
|
||||
|
||||
> [!question] 为什么 K8s 需要这么多组件?
|
||||
>
|
||||
> 因为"声明式 API"的设计哲学——用户告诉 K8s **想要什么状态**(比如"我要 3 个订单服务实例"),控制面负责让实际状态持续逼近目标状态。任何偏离都会被自动修复,这也就是自愈能力的来源。
|
||||
|
||||
### 核心对象速查表
|
||||
|
||||
| K8s 对象 | 用途 | 类比 |
|
||||
|---------|------|------|
|
||||
| **Pod** | 最小部署单元,包含一个或多个容器 | 应用实例 |
|
||||
| **Deployment** | 管理 Pod 的副本数和滚动更新 | 应用的"模板" |
|
||||
| **Service** | 稳定的网络入口,负载均衡 | 内部 VIP → [[03-网络与服务发现]] |
|
||||
| **ConfigMap** | 配置注入(明文) → [[02-配置管理]] |
|
||||
| **Secret** | 敏感配置注入(base64)→ [[02-配置管理]] |
|
||||
| **HPA** | 根据指标自动扩缩容 → [[04-扩缩容与有状态应用]] |
|
||||
| **StatefulSet** | 有状态应用的有序管理 → [[04-扩缩容与有状态应用]] |
|
||||
| **NetworkPolicy** | L3/L4 网络安全策略 → [[06-资源与安全管控]] |
|
||||
|
||||
## Deployment 详解
|
||||
|
||||
### 完整示例
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: order-service
|
||||
labels:
|
||||
app: order
|
||||
version: v1.2.3
|
||||
spec:
|
||||
replicas: 3 # 期望副本数
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxSurge: 1 # 最多超额 1 个 Pod
|
||||
maxUnavailable: 0 # 滚动更新期间不允许不可用
|
||||
|
||||
selector:
|
||||
matchLabels:
|
||||
app: order
|
||||
|
||||
template: # Pod 模板
|
||||
metadata:
|
||||
labels:
|
||||
app: order
|
||||
version: v1.2.3
|
||||
spec:
|
||||
containers:
|
||||
- name: order-service
|
||||
image: registry.example.com/order:v1.2.3
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
|
||||
# ========== 资源配置 ==========
|
||||
resources:
|
||||
requests: # 调度依据:保证至少有这些
|
||||
cpu: "250m"
|
||||
memory: "256Mi"
|
||||
limits: # 硬上限:超过则 OOMKill/CPU Throttle
|
||||
cpu: "500m"
|
||||
memory: "512Mi"
|
||||
|
||||
# ========== 探针 ==========
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: 8080
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 10
|
||||
failureThreshold: 3 # 连续失败 3 次才重启
|
||||
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: 8080
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: 8080
|
||||
failureThreshold: 30 # 最长等待 300s (慢启动友好)
|
||||
|
||||
# ========== 环境变量 & 挂载 ==========
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: order-service-config
|
||||
- secretRef:
|
||||
name: order-service-secrets
|
||||
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command: ["sh", "-c", "sleep 15"] # 优雅退出,给 LB 摘流时间
|
||||
```
|
||||
|
||||
### Probe 选择指南
|
||||
|
||||
| 探针类型 | 触发条件 | 动作 | 适用场景 |
|
||||
|---------|---------|------|---------|
|
||||
| **Liveness** | `/healthz` 返回非 2xx | 重启容器 | 死锁、无法恢复的崩溃 |
|
||||
| **Readiness** | `/ready` 返回非 2xx | 摘除 Service 流量 | 依赖未就绪、热加载中 |
|
||||
| **Startup** | 首次成功前持续失败 | 不重启,只等待 | 大模型/JVM 冷启动 |
|
||||
|
||||
> [!warning] 常见陷阱:Liveness 误杀导致 CrashLoopBackOff
|
||||
>
|
||||
> 如果 Liveness Probe 因为 DB 连接超时而返回 503,K8s 会认为容器挂了并反复重启它。正确做法:让 `/healthz` 做降级判断(DB 不可用时返回 200),用 `/ready` 来摘除流量。
|
||||
|
||||
### Deployment 更新策略
|
||||
|
||||
```yaml
|
||||
strategy:
|
||||
type: RollingUpdate # 逐步替换旧 Pod
|
||||
rollingUpdate:
|
||||
maxSurge: 1 # 新 Pod 数量可以超出期望值 1
|
||||
maxUnavailable: 0 # 更新时不允许有空缺
|
||||
|
||||
# 另一种策略:先全部新建再销毁旧的(零停机但需 2x 资源)
|
||||
# type: Recreate # ❌ 全量替换,会有短暂不可用
|
||||
```
|
||||
|
||||
> [!tip] 如何安全地回滚 Deployment?
|
||||
>
|
||||
> ```bash
|
||||
> kubectl rollout undo deployment/order-service -n production # 回到上一版本
|
||||
> kubectl rollout undo deployment/order-service -n production --to-revision=3 # 回退到指定版本
|
||||
> kubectl rollout status deployment/order-service -n production # 查看进度
|
||||
> kubectl rollout history deployment/order-service -n production # 历史版本对比
|
||||
> ```
|
||||
>
|
||||
> K8s 自动保留每次 Deployment 变更后的 PodTemplateSpec,因此回滚是瞬时的,不需要手动备份 YAML。
|
||||
|
||||
### Pod 生命周期简述
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Pending: Pod 被创建
|
||||
Pending --> ContainerCreating: 调度成功,拉取镜像
|
||||
ContainerCreating --> Running: 容器就绪
|
||||
Running --> Waiting: Probe 失败或手动暂停
|
||||
Waiting --> Running: 探针恢复
|
||||
Running --> Terminating: 删除/更新触发
|
||||
Terminating --> (*): 完全终止
|
||||
|
||||
state Waiting {
|
||||
CrashLoopBackOff
|
||||
ImagePullBackOff
|
||||
}
|
||||
|
||||
note right of CrashLoopBackOff: Liveness 连续失败\n或进程异常退出
|
||||
note right of ImagePullBackOff: 镜像不存在或仓库认证失败
|
||||
```
|
||||
|
||||
## 关联笔记
|
||||
|
||||
- [[../02-配置管理]] — ConfigMap / Secret 的配置注入
|
||||
- [[../03-网络与服务发现]] — Service / Ingress 网络路由
|
||||
- [[../04-扩缩容与有状态应用]] — HPA / StatefulSet
|
||||
- [[../hhs/MS/05-部署运维/01-容器化]] — Docker 镜像是 Pod 的基础单元
|
||||
- [[../hhs/MS/05-部署运维/03-CICD与GitOps]] — GitOps ArgoCD 操作 K8s manifest
|
||||
@@ -0,0 +1,188 @@
|
||||
---
|
||||
tags: [kubernetes, configmap, secret, configuration-management]
|
||||
create time: 2026-05-18 00:30
|
||||
---
|
||||
|
||||
# K8s 配置管理 — ConfigMap 与 Secret
|
||||
|
||||
## 概述
|
||||
|
||||
K8s 提供两种原生的配置注入机制:**ConfigMap**(普通配置)和 **Secret**(敏感信息)。理解它们的区别、注入方式和生效时机,是编写可维护 K8s manifest 的基础。更多安全方案(外部密钥管理)参见 [[../03-CICD与GitOps/04-安全与发布策略]]。
|
||||
|
||||
## ConfigMap — 非敏感配置注入
|
||||
|
||||
### 基本用法
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: order-service-config
|
||||
namespace: production
|
||||
data:
|
||||
# ========== 环境变量风格 ==========
|
||||
DATABASE_HOST: "mysql.production.svc.cluster.local"
|
||||
DATABASE_PORT: "3306"
|
||||
LOG_LEVEL: "info"
|
||||
MAX_CONNECTIONS: "100"
|
||||
|
||||
# ========== 文件风格 (挂载为 volume) ==========
|
||||
application.yaml: |
|
||||
server:
|
||||
port: 8080
|
||||
spring:
|
||||
datasource:
|
||||
url: jdbc:mysql://${DATABASE_HOST}:${DATABASE_PORT}/orders
|
||||
pool-size: ${MAX_CONNECTIONS}
|
||||
logging:
|
||||
level:
|
||||
root: ${LOG_LEVEL}
|
||||
```
|
||||
|
||||
### 两种注入方式
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
containers:
|
||||
- name: order-service
|
||||
# 方式一:环境变量注入(简单直接)
|
||||
env:
|
||||
- name: DATABASE_HOST
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: order-service-config
|
||||
key: DATABASE_HOST
|
||||
|
||||
# 方式二:volume 挂载配置文件(适合完整配置文件)
|
||||
volumeMounts:
|
||||
- name: config-volume
|
||||
mountPath: /etc/app/config/application.yaml
|
||||
subPath: application.yaml
|
||||
volumes:
|
||||
- name: config-volume
|
||||
configMap:
|
||||
name: order-service-config
|
||||
```
|
||||
|
||||
> [!tip] 热更新 vs 重建 Pod
|
||||
>
|
||||
> - 通过 **环境变量** 注入的配置需要重建 Pod 才能生效
|
||||
> - 通过 **volume 挂载** 的 ConfigMap 在配置变更后,Pod 内文件会在约 1 分钟内自动更新(前提是应用本身支持配置热加载,如 Spring Cloud Context 的 `@RefreshScope`)
|
||||
|
||||
## Secret — 敏感信息存储
|
||||
|
||||
### 基本用法
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: order-service-secrets
|
||||
namespace: production
|
||||
type: Opaque
|
||||
data:
|
||||
# Base64 编码(注意:这不是加密!只是编码)
|
||||
DB_PASSWORD: cGFzc3dvcmQxMjM=
|
||||
API_KEY: bXktc2VjcmV0LWFwaS1rZXk=
|
||||
|
||||
# 或者用 stringData(明文写入,K8s 自动编码)
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: order-service-secrets-v2
|
||||
type: Opaque
|
||||
stringData:
|
||||
DB_PASSWORD: password123 # K8s 自动转为 base64
|
||||
API_KEY: my-secret-api-key
|
||||
```
|
||||
|
||||
**使用 Secret 的方式:**
|
||||
|
||||
```yaml
|
||||
# 方式一:作为环境变量注入
|
||||
env:
|
||||
- name: DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: order-service-secrets
|
||||
key: DB_PASSWORD
|
||||
|
||||
# 方式二:挂载为卷中的文件(更安全,避免暴露在 ps 输出中)
|
||||
volumeMounts:
|
||||
- name: secrets-volume
|
||||
readOnly: true
|
||||
mountPath: /etc/secrets
|
||||
defaultMode: 0400
|
||||
volumes:
|
||||
- name: secrets-volume
|
||||
secret:
|
||||
secretName: order-service-secrets
|
||||
```
|
||||
|
||||
> [!warning] Secret 的安全性边界
|
||||
>
|
||||
> K8s Secret 默认以 **base64 编码** 存储在 etcd 中,任何人都可以解码。**Base64 ≠ 加密**。生产环境应启用以下至少一项:
|
||||
>
|
||||
> 1. **etcd 加密** — 开启 K8s EncryptionConfiguration,使 etcd 存储密文
|
||||
> 2. **外部密钥管理** — Sealed Secrets、External Secrets Operator 对接 AWS Secrets Manager / HashiCorp Vault
|
||||
> 3. **RBAC 严格管控** — 限制谁可以读取 Secret 资源
|
||||
|
||||
### ConfigMap vs Secret 对比
|
||||
|
||||
| 维度 | ConfigMap | Secret |
|
||||
|------|-----------|--------|
|
||||
| **用途** | 普通配置(URL、端口等) | 敏感信息(密码、Token、证书) |
|
||||
| **存储格式** | UTF-8 字符串 | Base64 编码 |
|
||||
| **挂载路径** | `/etc/config/...` | `/etc/secrets/...` |
|
||||
| **大小限制** | 1 MiB | 1 MiB |
|
||||
| **安全建议** | 正常 RBAC | 必须启用 etcd 加密或外置密钥管理 |
|
||||
|
||||
## 补充:动态配置重载实战
|
||||
|
||||
对于需要频繁变更的配置(如功能开关、灰度比例),可以通过 sidecar 模式实现自动重载:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
containers:
|
||||
- name: app
|
||||
image: myapp:latest
|
||||
volumeMounts:
|
||||
- name: config-volume
|
||||
mountPath: /etc/app/config
|
||||
readOnly: true
|
||||
- name: reload-sidecar # Watch 配置变更并发送 SIGUSR1
|
||||
image: busybox:latest
|
||||
command: ["sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
LAST_MD5=""
|
||||
while true; do
|
||||
CURR_MD5=$(md5sum /etc/config/application.yaml | awk '{print $1}')
|
||||
if [ "$CURR_MD5" != "$LAST_MD5" ]; then
|
||||
kill -USR1 1 # 通知主进程重新加载配置
|
||||
LAST_MD5="$CURR_MD5"
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
volumeMounts:
|
||||
- name: config-volume
|
||||
mountPath: /etc/config
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: config-volume
|
||||
configMap:
|
||||
name: order-service-config
|
||||
- name: shared-config
|
||||
emptyDir: {}
|
||||
```
|
||||
|
||||
> [!note] sidecar 模式的权衡
|
||||
>
|
||||
> 优点:无需重启 Pod 即可热更新配置;缺点:增加了资源消耗和维护复杂度。Spring Boot 项目建议使用 Actuator `/refresh` 端点替代手动信号方案。
|
||||
|
||||
## 关联笔记
|
||||
|
||||
- [[../01-核心概念与Deployment]] — ConfigMap 在 Deployment 中的标准引用方式
|
||||
- [[../03-CICD与GitOps/04-安全与发布策略]] — 企业级 Secret 管理方案选型
|
||||
- [[../hhs/MS/02-服务治理/07-配置管理]] — 微服务级别的配置中心方案(Nacos / Apollo / Consul)
|
||||
@@ -0,0 +1,144 @@
|
||||
---
|
||||
tags: [kubernetes, service, ingress, networking, service-discovery]
|
||||
create time: 2026-05-18 00:30
|
||||
---
|
||||
|
||||
# K8s 网络与服务发现 — Service 与 Ingress
|
||||
|
||||
## 概述
|
||||
|
||||
Kubernetes 的网络模型解决了容器 IP 频繁变动的问题。Service 提供**稳定的服务发现入口**,Ingress 提供**L7 HTTP 路由能力**。本文将梳理核心概念、典型模式和常见问题。
|
||||
|
||||
## Service 类型
|
||||
|
||||
| 类型 | 特点 | 使用场景 |
|
||||
|------|------|---------|
|
||||
| **ClusterIP** | 集群内 IP,外部不可访问 | 默认,内部服务间调用 |
|
||||
| **NodePort** | 在每个 Node 上开端口 | 调试、临时访问 |
|
||||
| **LoadBalancer** | 云厂商分配公网 IP | 对外暴露的服务 |
|
||||
| **ExternalName** | CNAME 到外部域名 | 对接外部系统 |
|
||||
|
||||
### ClusterIP Service — 服务发现的载体
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: order-service
|
||||
spec:
|
||||
selector:
|
||||
app: order # 匹配带此 label 的 Pod
|
||||
ports:
|
||||
- port: 80 # Service 端口(对外暴露)
|
||||
targetPort: 8080 # 容器实际监听端口
|
||||
protocol: TCP
|
||||
type: ClusterIP
|
||||
```
|
||||
|
||||
调用方只需 `http://order-service:80`,K8s 通过 iptables/IPVS 自动实现负载均衡。
|
||||
|
||||
> [!question] Service 是怎么做到负载均衡的?
|
||||
>
|
||||
> 每个 Node 上的 kube-proxy 会监控 Service 的变动,自动生成 iptables 规则或 IPVS 虚拟服务器配置。当请求发往 ClusterIP 时,内核将其 DNAT 到后端 Pod IP 之一,负载均衡算法默认是 round-robin。K8s 1.14+ 推荐使用 IPVS 模式(`kube-proxy --proxy-mode=ipvs`),性能更高且支持更多算法。
|
||||
|
||||
## Ingress — HTTP/HTTPS 路由
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: main-ingress
|
||||
annotations:
|
||||
nginx.ingress.kubernetes.io/rewrite-target: /
|
||||
spec:
|
||||
rules:
|
||||
- host: api.example.com
|
||||
http:
|
||||
paths:
|
||||
- path: /orders
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: order-service
|
||||
port:
|
||||
number: 80
|
||||
- path: /users
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: user-service
|
||||
port:
|
||||
number: 80
|
||||
```
|
||||
|
||||
> [!info] Ingress Controller 是什么?
|
||||
>
|
||||
> Ingress 只是一个 API 对象,真正的 HTTP 路由由 Ingress Controller(如 NGINX、Traefik、Envoy)执行。安装后会在集群中运行一个 LoadBalancer 类型的 Pod 集群,监听 Ingress 资源变化并生成对应配置。
|
||||
|
||||
### 常见 Ingress Annotation(NGINX 为例)
|
||||
|
||||
| Annotation | 用途 | 示例值 |
|
||||
|------------|------|--------|
|
||||
| `rewrite-target` | URL 重写 | `/` |
|
||||
| `ssl-redirect` | 强制 HTTPS | `"true"` |
|
||||
| `rate-limit` | 限流 | `100` |
|
||||
| `whitelist-source-range` | IP 白名单 | `10.0.0.0/8` |
|
||||
| `client-max-body-size` | 上传文件大小限制 | `50m` |
|
||||
|
||||
## Headless Service — 无头服务
|
||||
|
||||
Headless Service (`ClusterIP: None`) 不分配虚拟 IP,而是返回所有匹配 Pod 的真实 IP。这是 StatefulSet 的灵魂搭档:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: mysql-headless
|
||||
spec:
|
||||
clusterIP: None # 关键:不分配 ClusterIP
|
||||
selector:
|
||||
app: mysql
|
||||
ports:
|
||||
- port: 3306
|
||||
```
|
||||
|
||||
通过 DNS 可以直接访问单个 Pod:
|
||||
- `mysql-cluster-0.mysql-headless.production.svc.cluster.local:3306`
|
||||
- `mysql-cluster-1.mysql-headless.production.svc.cluster.local:3306`
|
||||
|
||||
## K8s 网络模型总结
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
EXT["外部流量"] --> INGRESS["Ingress Controller"]
|
||||
|
||||
INGRESS --> SVC_A["order-service:80"]
|
||||
INGRESS --> SVC_B["user-service:80"]
|
||||
|
||||
SVC_A --> Pod1["order-pod-1\n10.244.1.5:8080"]
|
||||
SVC_A --> Pod2["order-pod-2\n10.244.2.3:8080"]
|
||||
|
||||
SVC_B --> Pod3["user-pod-1\n10.244.1.8:3000"]
|
||||
|
||||
Pod1 -->|DNS解析| COREDNS["CoreDNS<br/>svc.cluster.local"]
|
||||
|
||||
style EXT fill:#e3f2fd
|
||||
style INGRESS fill:#fff3e0
|
||||
style SVC_A fill:#e8f5e9
|
||||
style SVC_B fill:#fce4ec
|
||||
```
|
||||
|
||||
> [!tip] 调试技巧:Service 不通怎么办?
|
||||
>
|
||||
> 1. `kubectl get svc <name>` 确认 Service 存在且有 ClusterIP
|
||||
> 2. `kubectl get endpoints <name>` 检查 Endpoint 列表是否为空
|
||||
> 3. Endpoint 为空 → 检查 selector 标签是否匹配 Pod
|
||||
> 4. Endpoint 有值但 curl 不通 → 进入 Pod `curl <ClusterIP>:<port>` 验证
|
||||
> 5. ClusterIP 通但外部不通 → 检查 Ingress / LoadBalancer 配置
|
||||
|
||||
## 关联笔记
|
||||
|
||||
- [[../01-核心概念与Deployment]] — Deployment 通过 selector 与 Service 关联
|
||||
- [[../02-配置管理]] — Service 的端点配置不直接涉及 ConfigMap/Secret
|
||||
- [[../04-扩缩容与有状态应用]] — Headless Service + StatefulSet 配合
|
||||
- [[../hhs/MS/02-服务治理/04-服务发现]] — K8s Service 是服务端发现模式的代表
|
||||
@@ -0,0 +1,217 @@
|
||||
---
|
||||
tags: [kubernetes, hpa, autoscaling, statefulset, horizontal-pod-autoscaler]
|
||||
create time: 2026-05-18 00:30
|
||||
---
|
||||
|
||||
# K8s 扩缩容与有状态应用 — HPA & StatefulSet
|
||||
|
||||
## 概述
|
||||
|
||||
K8s 提供多种扩缩容机制。**HPA** 基于 CPU/Memory 或其他自定义指标实现 Pod 级别的弹性伸缩,适用于无状态服务。**StatefulSet** 则是数据库、消息队列等有状态组件的理想选择。两者各有适用场景。
|
||||
|
||||
## HPA 弹性伸缩
|
||||
|
||||
### 基础示例
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: order-service-hpa
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: order-service
|
||||
minReplicas: 3
|
||||
maxReplicas: 20
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70 # CPU > 70% 时扩容
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 80
|
||||
behavior:
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 60 # 扩容稳定期
|
||||
policies:
|
||||
- type: Pods
|
||||
value: 2
|
||||
periodSeconds: 60 # 每分钟最多扩 2 个
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 300 # 缩容稳定期 5min(防抖动)
|
||||
```
|
||||
|
||||
### 扩展指标:基于自定义 Metric 扩缩容
|
||||
|
||||
```yaml
|
||||
metrics:
|
||||
- type: Pods
|
||||
pods:
|
||||
metric:
|
||||
name: http_requests_per_second
|
||||
target:
|
||||
type: AverageValue
|
||||
averageValue: "100" # 每个 Pod 平均 100 QPS 时扩容
|
||||
- type: Object
|
||||
object:
|
||||
metric:
|
||||
name: active_users
|
||||
describedObject:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: web-app
|
||||
target:
|
||||
type: Value
|
||||
value: "500" # 每 500 活跃用户触发一次扩容
|
||||
```
|
||||
|
||||
> [!info] HPA 的工作机制
|
||||
>
|
||||
> HPA 控制器每 15 秒查询 Metrics Server(CPU/Memory)或自定义 Metrics Adapter(QPS 等),计算所需 Replica 数。公式:`ceil(当前指标值 / 目标指标值 × 当前副本数)`。为了防止抖动,scaleDown 的稳定窗口通常比 scaleUp 长很多。
|
||||
|
||||
### HPA 调优最佳实践
|
||||
|
||||
| 参数 | 推荐值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `averageUtilization` (CPU) | 60~70% | 低于 60% 浪费资源,高于 80% 延迟风险大 |
|
||||
| `scaleUp.stabilizationWindow` | 0~60s | 突发流量快速响应 |
|
||||
| `scaleDown.stabilizationWindow` | 300~600s | 防止抖动导致的频繁扩缩 |
|
||||
| `pods[].periodSeconds` | 60s | 限制扩容频率,避免雪崩 |
|
||||
|
||||
## StatefulSet — 有状态应用
|
||||
|
||||
Deployment 适合无状态服务,而有状态服务(数据库、分布式协调器等)需要 **稳定的网络标识 + 有序的启停**。StatefulSet 就是为此而生的控制器。
|
||||
|
||||
### 完整示例
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: mysql-cluster
|
||||
spec:
|
||||
serviceName: mysql # 绑定 Headless Service
|
||||
replicas: 3
|
||||
podManagementPolicy: Parallel # 并行启动(生产建议 OrderedReady 逐步启动)
|
||||
|
||||
updateStrategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
partition: 0 # 从最后一个 Pod 开始逐步滚动
|
||||
|
||||
selector:
|
||||
matchLabels:
|
||||
app: mysql
|
||||
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: mysql
|
||||
spec:
|
||||
containers:
|
||||
- name: mysql
|
||||
image: mysql:8.0
|
||||
ports:
|
||||
- containerPort: 3306
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /var/lib/mysql
|
||||
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: data
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
resources:
|
||||
requests:
|
||||
storage: 20Gi
|
||||
```
|
||||
|
||||
### StatefulSet 的核心特性
|
||||
|
||||
| 特性 | 说明 |
|
||||
|------|------|
|
||||
| **稳定名称** | Pod 名为 `{statefulset-name}-{ordinal}`,如 `mysql-cluster-0` |
|
||||
| **有序部署** | 默认依次启动:0 → 1 → 2(设置 `Parallel` 可并行) |
|
||||
| **有序删除** | 反向删除:2 → 1 → 0(保证主从切换安全) |
|
||||
| **持久化存储** | `volumeClaimTemplates` 为每个 Pod 独立创建 PVC,Pod 重建后数据不丢失 |
|
||||
| **Headless Service** | 需搭配 `ClusterIP: None` 的 Service,每个 Pod 有独立的 DNS 记录 |
|
||||
|
||||
### StatefulSet vs Deployment 对比
|
||||
|
||||
| 维度 | Deployment | StatefulSet |
|
||||
|------|-----------|-------------|
|
||||
| **Pod 命名** | 随机后缀 | `{name}-{0,1,2,...}` |
|
||||
| **存储** | 共享 PVC 或不持久 | 每个 Pod 独立 PVC |
|
||||
| **启动顺序** | 并发 | 串行(默认) |
|
||||
| **删除顺序** | 随机 | 逆序 |
|
||||
| **DNS 稳定性** | 每次重建 IP 变 | 域名不变 |
|
||||
| **适用场景** | Web/API 无状态服务 | DB、ZooKeeper、Kafka |
|
||||
|
||||
> [!note] 为什么有状态应用不适合 Deployment?
|
||||
>
|
||||
> Deployment 会随机销毁和重建 Pod——这意味着 PVC 会被重新绑定到不同的 Pod,存储卷中的数据和节点 ID 完全错位。StatefulSet 保证同一个 Pod 始终使用同一片存储,这对数据库、Kafka、ZooKeeper 等组件至关重要。
|
||||
|
||||
### Headless Service — StatefulSet 的灵魂搭档
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: mysql-headless
|
||||
spec:
|
||||
clusterIP: None # 关键:不分配 ClusterIP
|
||||
selector:
|
||||
app: mysql
|
||||
ports:
|
||||
- port: 3306
|
||||
```
|
||||
|
||||
通过 DNS 可以直接访问单个 Pod:
|
||||
- `mysql-cluster-0.mysql-headless.production.svc.cluster.local:3306`
|
||||
- `mysql-cluster-1.mysql-headless.production.svc.cluster.local:3306`
|
||||
|
||||
## 补充:Vertical Pod Autoscaler (VPA)
|
||||
|
||||
HPA 只管 Pod 数量,不管单 Pod 的资源配置。如果你经常遇到 OOMKill 或 CPU Throttling,可以尝试 VPA:
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling.k8s.io/v1
|
||||
kind: VerticalPodAutoscaler
|
||||
metadata:
|
||||
name: order-service-vpa
|
||||
spec:
|
||||
targetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: order-service
|
||||
updatePolicy:
|
||||
updateMode: "Auto" # Auto / Initial / Off
|
||||
resourcePolicy:
|
||||
containerPolicy:
|
||||
minAllowed:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
maxAllowed:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
```
|
||||
|
||||
> [!warning] VPA 的限制
|
||||
>
|
||||
> VPA 在调整资源时会**驱逐并重建 Pod**。与 HPA 不同,VPA 不能无缝运行——建议在开发/测试环境中先用 `Initial` 模式观察推荐值,确定合理后再切到 `Auto`。生产环境通常结合 HPA(管数量)+ 手动调优 Resources(管质量)。
|
||||
|
||||
## 关联笔记
|
||||
|
||||
- [[../01-核心概念与Deployment]] — Deployment 基础,HPA 的目标控制器
|
||||
- [[../02-配置管理]] — StatefulSet 的数据库凭据通过 Secret 注入
|
||||
- [[../03-网络与服务发现]] — Headless Service 为 StatefulSet 提供 DNS 解析
|
||||
- [[../hhs/MS/05-部署运维/04-SRE实践]] — SLO/Error Budget 决定扩容阈值
|
||||
@@ -0,0 +1,179 @@
|
||||
---
|
||||
tags: [kubernetes, scheduling, affinity, taints, tolerations, topology]
|
||||
create time: 2026-05-18 00:30
|
||||
---
|
||||
|
||||
# K8s 调度控制 — Affinity、Taint & Topology
|
||||
|
||||
## 概述
|
||||
|
||||
默认调度器会把 Pod 随便分配到任意可用 Node。想让 Pod 之间"互相躲开"或"必须在一起"?需要使用 K8s 的调度控制机制:**Affinity/Anti-Affinity**(节点间关系)、**Taint/Toleration**(节点白名单)、**TopologySpreadConstraints**(跨可用区分布)。
|
||||
|
||||
## Affinity / Anti-Affinity
|
||||
|
||||
### Pod 反亲和性:分散到不同节点
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
affinity:
|
||||
# ========== 反亲和性:同一服务的 Pod 分散到不同节点 ==========
|
||||
podAntiAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 100
|
||||
podAffinityTerm:
|
||||
labelSelector:
|
||||
matchExpressions:
|
||||
- key: app
|
||||
operator: In
|
||||
values:
|
||||
- order-service
|
||||
topologyKey: kubernetes.io/hostname
|
||||
# ^^^ "尽量"把同 app 的 Pod 分配到不同机器
|
||||
```
|
||||
|
||||
> [!question] 为什么要把同一个服务的 Pod 分散?
|
||||
>
|
||||
> 如果所有副本都在同一台物理机上,这台机器宕机就会导致服务完全不可用。**故障域隔离**是高可用的基础——把 Pod 分散到不同节点 = 把鸡蛋放进不同篮子。
|
||||
|
||||
### 节点亲和性:只调度到特定标签的节点
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
affinity:
|
||||
# ========== 节点亲和性:只调度到特定标签的节点 ==========
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: node-role
|
||||
operator: In
|
||||
values:
|
||||
- high-memory # 只调度到有 high-memory 标签的节点
|
||||
```
|
||||
|
||||
### 调度级别对比
|
||||
|
||||
| 级别 | 含义 | 推荐度 |
|
||||
|---------|------|--------|
|
||||
| `requiredDuring...` | 硬约束,不满足就不调度 | 高(适用于特殊硬件节点) |
|
||||
| `preferredDuring...` | 软约束,尽力而为 | 高(适用于反亲和性拆分) |
|
||||
| `ignoringDuringExecution` | 调度时检查,运行时忽略变化 | 标准做法 |
|
||||
|
||||
## Taint & Toleration — 节点级别的"白名单"
|
||||
|
||||
Taint 设在 Node 上,Toleration 设在 Pod 上:
|
||||
|
||||
```bash
|
||||
# 给节点加污点:只有带 db 容忍度的 Pod 才能调度上来
|
||||
kubectl taint nodes node1 dedicated=db:NoSchedule
|
||||
```
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
tolerations:
|
||||
- key: "dedicated"
|
||||
operator: "Equal"
|
||||
value: "db"
|
||||
effect: "NoSchedule"
|
||||
```
|
||||
|
||||
> [!info] Taint effect 三种类型
|
||||
>
|
||||
> | Effect | 含义 | 场景 |
|
||||
> |--------|------|------|
|
||||
> | `NoSchedule` | 不接受新 Pod | 专用节点 |
|
||||
> | `PreferNoSchedule` | 尽量避免但不强制 | 优先但不是硬性 |
|
||||
> | `NoExecute` | 驱逐已有的 Pod | 节点出问题时 |
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Node["Node 打 Taint<br/>dedicated=db:NoSchedule"] --> Check{Pod 有 Toleration?}
|
||||
|
||||
Check -- 否 --> Reject["❌ 不允许调度"]
|
||||
Check -- 是 --> Accept["✅ 允许调度"]
|
||||
|
||||
style Reject fill:#ffebee
|
||||
style Accept fill:#e8f5e9
|
||||
```
|
||||
|
||||
## TopologySpreadConstraints — K8s 1.19+ 推荐的跨可用区分布
|
||||
|
||||
比 anti-affinity 更优雅的实现方式,特别适合多云和混合云架构:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1 # 任何两个可用区最多差 1 个 Pod
|
||||
topologyKey: topology.kubernetes.io/zone # 按可用区分组
|
||||
whenUnsatisfiable: DoNotSchedule # 不满足则不调度
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: order-service
|
||||
```
|
||||
|
||||
> [!tip] WhenUnsatisfiable 策略选择
|
||||
>
|
||||
> - `DoNotSchedule`:宁可等也不破坏分布均衡(严格一致)
|
||||
> - `ScheduleAnyway`:先调度上去,能分尽量分(可用性优先)
|
||||
>
|
||||
> **生产建议**:对外服务用 `ScheduleAnyway` + 反亲和性;内部有状态组件用 `DoNotSchedule`。
|
||||
|
||||
## 补充:NodeSelector 最简单的限制
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
nodeSelector:
|
||||
disktype: ssd # 只调度到标注了 disktype=ssd 的节点
|
||||
```
|
||||
|
||||
> [!note] NodeSelector vs Node Affinity
|
||||
>
|
||||
> NodeSelector 是最基本的调度方式——只能做相等匹配。如果需要 `In/NotIn/Exists` 等更复杂的逻辑,应使用 `nodeAffinity`。现代 K8s 项目中建议统一使用 `nodeAffinity` 以保持风格一致。
|
||||
|
||||
## 组合使用示例
|
||||
|
||||
```yaml
|
||||
# 一个典型的 Production Web 部署
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
# ① 必须运行在 Linux x86_64 节点上
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: kubernetes.io/os
|
||||
operator: In
|
||||
values: [linux]
|
||||
- key: kubernetes.io/arch
|
||||
operator: In
|
||||
values: [amd64]
|
||||
|
||||
# ② 尽量分散在不同可用区
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: topology.kubernetes.io/zone
|
||||
whenUnsatisfiable: ScheduleAnyway
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: web-frontend
|
||||
|
||||
# ③ 避免与后台批处理任务混在同一节点
|
||||
podAntiAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 200
|
||||
podAffinityTerm:
|
||||
labelSelector:
|
||||
matchExpressions:
|
||||
- key: workload-type
|
||||
operator: In
|
||||
values: [batch-job]
|
||||
topologyKey: kubernetes.io/hostname
|
||||
```
|
||||
|
||||
## 关联笔记
|
||||
|
||||
- [[../01-核心概念与Deployment]] — Deployment 中的 podTemplateSpec 位置
|
||||
- [[../06-资源与安全管控]] — ResourceQuota / NetworkPolicy 配合调度实现安全隔离
|
||||
- [[../hhs/MS/05-部署运维/02-Kubernetes]] — 调度失控时的常见排查技巧
|
||||
@@ -0,0 +1,208 @@
|
||||
---
|
||||
tags: [kubernetes, resource-quota, limit-range, network-policy, security]
|
||||
create time: 2026-05-18 00:30
|
||||
---
|
||||
|
||||
# K8s 资源与安全管控 — Quota、LimitRange & NetworkPolicy
|
||||
|
||||
## 概述
|
||||
|
||||
多人共享集群或多租户环境中,需要两道防线来保障稳定:**资源管控**防止单个团队耗尽集群,**网络策略**防止横向越权访问。这两者构成了 K8s 多环境的安全基线。
|
||||
|
||||
## ResourceQuota — 命名空间资源总量上限
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ResourceQuota
|
||||
metadata:
|
||||
name: production-quota
|
||||
namespace: production
|
||||
spec:
|
||||
hard:
|
||||
requests.cpu: "16" # 所有 Pod 的 CPU request 总和不超过 16核
|
||||
requests.memory: 32Gi # 内存 request 不超过 32G
|
||||
limits.cpu: "32" # 所有 Pod 的 CPU limit 不超过 32核
|
||||
limits.memory: 64Gi # 内存 limit 不超过 64G
|
||||
pods: "50" # 最多 50 个 Pod
|
||||
services: "20" # 最多 20 个 Service
|
||||
persistentvolumeclaims: "10" # 最多 10 个 PVC
|
||||
```
|
||||
|
||||
> [!warning] 配额打满怎么办?
|
||||
>
|
||||
> 当某个命名空间的配额被占满时,该 namespace 内的新创建请求会直接失败(Return Error)。解决方法:清理不再使用的资源,或申请管理员增加配额 (`kubectl edit quota <name> -n <namespace>`)。
|
||||
|
||||
## LimitRange — 单容器资源边界
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: LimitRange
|
||||
metadata:
|
||||
name: default-limits
|
||||
namespace: production
|
||||
spec:
|
||||
limits:
|
||||
# 默认 Limits/Requests(不指定的 Pod 自动套用)
|
||||
- type: Container
|
||||
default:
|
||||
cpu: "500m"
|
||||
memory: "512Mi"
|
||||
defaultRequest:
|
||||
cpu: "250m"
|
||||
memory: "256Mi"
|
||||
# 单个容器的边界
|
||||
max:
|
||||
cpu: "2"
|
||||
memory: "4Gi"
|
||||
min:
|
||||
cpu: "50m"
|
||||
memory: "64Mi"
|
||||
```
|
||||
|
||||
> [!info] Quota vs LimitRange — 别搞混
|
||||
>
|
||||
> - **ResourceQuota** = namespace 总量天花板(整个房间能住多少人)
|
||||
> - **LimitRange** = 单容器边界(每人占多大床位)
|
||||
>
|
||||
> 两者配合使用,既能防止个体乱配资源,又能防止群体耗尽配额。
|
||||
|
||||
## NetworkPolicy — 网络安全策略
|
||||
|
||||
默认情况下,K8s 集群内所有 Pod 可以自由互访(零信任缺失)。NetworkPolicy 提供 L3/L4 层的双向流量控制:
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: order-service-policy
|
||||
namespace: production
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: order-service
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
|
||||
ingress:
|
||||
# 允许来自 Ingress Controller 和 user-service 的流量
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
name: ingress-ns
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: user-service
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
|
||||
egress:
|
||||
# 允许 outbound DB 连接和 DNS 查询
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: mysql
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3306
|
||||
- to:
|
||||
- namespaceSelector: {}
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53 # DNS(必备!否则 Pod 无法解析域名)
|
||||
```
|
||||
|
||||
### NetworkPolicy 最佳实践
|
||||
|
||||
| 原则 | 说明 |
|
||||
|------|------|
|
||||
| **默认拒绝全部** | 先用 `podSelector: {}` + 空 ingress/egress 拦截一切 |
|
||||
| **白名单精确匹配** | 每个 Policy 只放开一个方向的流量 |
|
||||
| **DNS 永远放行** | 没有 egress DNS 规则,Service 发现全部失效 |
|
||||
| **跨命名空间要显式声明** | namespaceSelector 为空 `{}` 表示允许所有命名空间 |
|
||||
|
||||
### 支持 NetworkPolicy 的网络插件
|
||||
|
||||
> [!warning] NetworkPolicy 生效前提
|
||||
>
|
||||
> NetworkPolicy **需要集群网络插件支持**。默认的 Kubenet / Flannel 不一定支持,推荐使用:
|
||||
> - **Calico** — 功能最完整,支持 L7 策略(与 Istio 配合)
|
||||
> - **Cilium** — 基于 eBPF,性能最优,原生支持 L7 过滤
|
||||
> - **Antrea** — VMware 出品,兼容性好
|
||||
|
||||
### 补充:Zero-Trust 默认策略模板
|
||||
|
||||
```yaml
|
||||
# ========== 全局:默认拒绝所有入站 ==========
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny-all-ingress
|
||||
namespace: production
|
||||
spec:
|
||||
podSelector: {} # 选择所有 Pod
|
||||
policyTypes:
|
||||
- Ingress
|
||||
|
||||
---
|
||||
# ========== 全局:默认拒绝所有出站 ==========
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny-all-egress
|
||||
namespace: production
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Egress
|
||||
```
|
||||
|
||||
> [!tip] 渐进式实施 NetworkPolicy
|
||||
>
|
||||
> 不要在已有集群上一夜之间切换所有策略——这会立刻打飞所有服务。正确的做法:
|
||||
>
|
||||
> 1. 先以 **audit** 模式运行 Calico/Cilium,观察现有流量
|
||||
> 2. 对每个服务编写 allowlist Policy,在 **测试环境** 验证
|
||||
> 3. 逐步推广到生产,从非关键服务开始
|
||||
> 4. 最终启用 deny-all 作为兜底
|
||||
|
||||
## 补充:SecurityContext — 容器级安全加固
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
containers:
|
||||
- name: order-service
|
||||
securityContext:
|
||||
runAsNonRoot: true # 禁止 root 运行
|
||||
readOnlyRootFilesystem: true # 根文件系统只读
|
||||
allowPrivilegeEscalation: false # 禁止提权
|
||||
capabilities:
|
||||
drop: ["ALL"] # 丢弃所有内核能力
|
||||
add: ["NET_BIND_SERVICE"] # 仅保留绑定 <1024 端口需要的能力
|
||||
|
||||
# Pod 级别的 DNS 配置
|
||||
dnsPolicy: ClusterFirstWithHostNet # DNS 策略
|
||||
automountServiceAccountToken: false # 不挂载 SA Token(不需要 API 时)
|
||||
```
|
||||
|
||||
> [!warning] readOnlyRootFilesystem 的坑
|
||||
>
|
||||
> 某些应用(如 Python 的 pip 缓存、Java 的 temp 目录)会尝试写入 `/tmp` 或应用目录。解决方案:
|
||||
>
|
||||
> ```yaml
|
||||
> volumeMounts:
|
||||
> - name: tmp-volume
|
||||
> mountPath: /tmp
|
||||
> volumes:
|
||||
> - name: tmp-volume
|
||||
> emptyDir: {}
|
||||
> ```
|
||||
>
|
||||
> 通过 `emptyDir` 挂载临时卷,既保持了根文件系统只读,又给了应用必要的写权限。
|
||||
|
||||
## 关联笔记
|
||||
|
||||
- [[../02-配置管理]] — Secret 资源与 RBAC 共同构成安全基线
|
||||
- [[../05-调度控制]] — 调度控制 + 网络策略 = 纵深防御
|
||||
- [[../03-CICD与GitOps/04-安全与发布策略]] — GitOps 中的 Secret 安全
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
tags: [kubernetes, troubleshooting, diagnostics, ops-checklist]
|
||||
create time: 2026-05-18 00:30
|
||||
---
|
||||
|
||||
# K8s 运维排查 — Checklist 与诊断速查
|
||||
|
||||
## 概述
|
||||
|
||||
每次上线前过一遍清单,遇到问题时有系统化的排查思路。本章整理了实战中最常用的诊断命令、高频问题对照表和有价值的调试技巧。更多 SRE 理念(错误预算、MTTR)参见 [[../04-SRE实践]]。
|
||||
|
||||
## 上线前 Checklist
|
||||
|
||||
| # | 检查项 | 说明 |
|
||||
|---|--------|------|
|
||||
| 1 | **Probe 已配置** | liveness/readiness/startup 都设定了阈值 |
|
||||
| 2 | **Resources Limits** | 防止单个 Pod OOMKill 拖垮整台机器 |
|
||||
| 3 | **日志输出到 stdout/stderr** | 可被采集器解析为 JSON |
|
||||
| 4 | **trace_id 透传** | 跨服务调用链 trace_id 不丢失 |
|
||||
| 5 | **回滚预案** | `kubectl rollout undo deployment/order-service` 能用 |
|
||||
| 6 | **告警已配置** | 关键指标异常时有人收到通知 |
|
||||
| 7 | **镜像 Tag** | 不用 latest,用语义化版本或 commit SHA |
|
||||
|
||||
## 常用诊断命令
|
||||
|
||||
```bash
|
||||
# 1. 查看 Pod 状态(为什么起不来?)
|
||||
kubectl get pods -n production
|
||||
|
||||
# 2. 看单个 Pod 的详细事件
|
||||
kubectl describe pod <pod-name> -n production
|
||||
# ↑ 重点看 Events 区域的 LastState / State / Reason
|
||||
|
||||
# 3. 看容器日志(含重启前的上一次输出)
|
||||
kubectl logs <pod-name> -n production --previous # 上次崩溃容器的日志
|
||||
kubectl logs <pod-name> -n production -c sidecar # 多容器指定 sidecar 名
|
||||
|
||||
# 4. 进入运行中的容器调试
|
||||
kubectl exec -it <pod-name> -n production -- /bin/sh
|
||||
|
||||
# 5. 查看滚动更新进度
|
||||
kubectl rollout status deployment/order-service -n production
|
||||
|
||||
# 6. 回滚到上一个版本
|
||||
kubectl rollout undo deployment/order-service -n production
|
||||
|
||||
# 7. 查看资源占用
|
||||
kubectl top pods -n production # Pod CPU/Memory
|
||||
kubectl top nodes # 节点级别
|
||||
```
|
||||
|
||||
## 高频问题对照表
|
||||
|
||||
| 症状 | 可能原因 | 排查步骤 |
|
||||
|------|---------|---------|
|
||||
| `ImagePullBackOff` | 镜像不存在、仓库认证失败、拼写错误 | `kubectl describe pod` 看 Normal Events;确认 registry 凭证 Secret |
|
||||
| `ErrImagePull` | 镜像 tag 不存在 | 检查 CI 是否成功 push;`docker pull` 在本地复现 |
|
||||
| `CrashLoopBackOff` | Liveness Probe 误杀、代码异常启动 | `kubectl logs --previous`;检查 `/healthz` 健康端点逻辑 |
|
||||
| `Pending` (调度中) | 资源不足、Affinity/Toleration 不满足 | `kubectl describe pod` 看 Warning 事件;检查节点可用资源 |
|
||||
| `OOMKilled` | memory limit 过小或内存泄漏 | 增大 limits;检查应用堆dump;JVM 需设 `-Xmx` |
|
||||
| Service 不通 | Selector 标签不匹配、端口配置错 | `kubectl get ep <svc>` 看 Endpoint 列表;curl ClusterIP 验证 |
|
||||
| ConfigMap/Secret 未生效 | 重建了 Pod 但环境变量没更新 | 删除 Pod 让 Deployment 重建;ConfigMap volume 挂载会热更新 |
|
||||
| Ingress 无响应 | Ingress Controller 未安装、Backend 配置错 | `kubectl get pods -n ingress-nginx`;检查 annotation 语法 |
|
||||
| DNS 解析失败 | CoreDNS Pod 异常或 NetworkPolicy 限制 DNS egress | `nslookup kubernetes.default`;确保 DNS egress 端口 53 放行 |
|
||||
| Pod 被频繁驱逐 | Node 资源不足触发 Eviction | `kubectl describe node <node>` 看 MemoryPressure/DiskPressure |
|
||||
|
||||
## 调试技巧:优雅地抓包与断点
|
||||
|
||||
```bash
|
||||
# ========== Debugging Sidecar 模式 ==========
|
||||
# 给故障 Pod 附加一个临时调试容器,共享网络命名空间
|
||||
kubectl debug <pod-name> -it --image=nicolaka/netshoot --share-network
|
||||
|
||||
# 进来了之后可以直接:
|
||||
# curl, nslookup, tcpdump, ping, tshark — 全套网络诊断工具
|
||||
|
||||
# ========== 动态调整日志级别(无需重建 Pod)==========
|
||||
# 通过 port-forward 访问 kube-apiserver 的 debug endpoint
|
||||
kubectl port-forward svc/kube-apiserver 6443:443 -n default
|
||||
```
|
||||
|
||||
> [!tip] 快速判断 K8s 问题的层级
|
||||
>
|
||||
> ```
|
||||
> Pod 起不来 → 查 Images / Resources / Probes
|
||||
> Pod 起来了但服务不通 → 查 Service Selector / Endpoints / Ingress
|
||||
> 服务通但有报错 → 查 App Logs / Metrics / Traces
|
||||
> 性能差 → 查 CPU Throttling / Disk IO / 连接池
|
||||
> ```
|
||||
>
|
||||
> 按这个顺序一层层定位,避免在日志里大海捞针。
|
||||
|
||||
## 补充:kubectl 高阶用法
|
||||
|
||||
```bash
|
||||
# 根据表达式筛选(比如只看处于 CrashLoop 的 Pod)
|
||||
kubectl get pods --field-selector=status.phase==Failed -A
|
||||
|
||||
# 批量执行命令(在每个 Pod 中同时运行)
|
||||
kubectl exec deploy/api-server -- sh -c 'uptime; free -m'
|
||||
|
||||
# 导出资源配置用于备份或审计
|
||||
kubectl get deployment -n production -o yaml > deploy-backup.yaml
|
||||
|
||||
# 模拟变更效果(dry-run,不做实际修改)
|
||||
kubectl apply -f new-deployment.yaml --dry-run=server -o yaml
|
||||
|
||||
# 查看哪个节点承载了某个 Pod
|
||||
kubectl get pods -o wide -n production | grep my-app
|
||||
|
||||
# 持续监控 Pod 事件(实时流)
|
||||
kubectl get events -n production --sort-by=.lastTimestamp -w
|
||||
```
|
||||
|
||||
> [!info] 理解 kubectl verbosity 级别
|
||||
>
|
||||
> `-v=6` 显示 HTTP 请求 headers;`-v=8` 额外返回响应 body;`-v=9` 逐行展开。调试 API 交互时通常 `-v=6` 就足够了,`-v=8` 以上会产生大量输出。
|
||||
|
||||
## 补充:集群层面的健康检查
|
||||
|
||||
```bash
|
||||
# 查看所有控制面组件状态
|
||||
kubectl get componentstatuses # K8s 1.19+ 已废弃,改用:
|
||||
kubectl get endpoints etcd -n kube-system
|
||||
|
||||
# 检查 CoreDNS 健康(DNS 异常的起点)
|
||||
kubectl get pods -n kube-system -l k8s-app=kube-dns
|
||||
kubectl logs -n kube-system <coredns-pod-name>
|
||||
|
||||
# 检查存储插件正常
|
||||
kubectl get storageclass
|
||||
|
||||
# 查看当前活跃的资源配额使用情况
|
||||
kubectl describe quota -n production
|
||||
```
|
||||
|
||||
## 关联笔记
|
||||
|
||||
- [[../01-核心概念与Deployment]] — Deployment 回滚操作
|
||||
- [[../03-网络与服务发现]] — Service / Ingress 的调试方法
|
||||
- [[../05-调度控制]] — Pending Pod 与 Affinity/Toleration 的关系
|
||||
- [[../hhs/MS/05-部署运维/04-SRE实践]] — MTTR 指标与故障恢复
|
||||
- [[../hhs/MS/04-可观测性]] — Prometheus + Grafana 可视化排查
|
||||
Reference in New Issue
Block a user