180 lines
5.6 KiB
Markdown
180 lines
5.6 KiB
Markdown
|
---
|
||
|
|
tags: [kubernetes, scheduling, affinity, taints, tolerations, topology]
|
||
|
|
create time: 2026-05-18 00:30
|
||
|
|
---
|
||
|
|
|
||
|
|
# K8s 调度控制 — Affinity、Taint & Topology
|
||
|
|
|
||
|
|
## 概述
|
||
|
|
|
||
|
|
默认调度器会把 Pod 随便分配到任意可用 Node。想让 Pod 之间"互相躲开"或"必须在一起"？需要使用 K8s 的调度控制机制：**Affinity/Anti-Affinity**（节点间关系）、**Taint/Toleration**（节点白名单）、**TopologySpreadConstraints**（跨可用区分布）。
|
||
|
|
|
||
|
|
## Affinity / Anti-Affinity
|
||
|
|
|
||
|
|
### Pod 反亲和性：分散到不同节点
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
spec:
|
||
|
|
affinity:
|
||
|
|
# ========== 反亲和性：同一服务的 Pod 分散到不同节点 ==========
|
||
|
|
podAntiAffinity:
|
||
|
|
preferredDuringSchedulingIgnoredDuringExecution:
|
||
|
|
- weight: 100
|
||
|
|
podAffinityTerm:
|
||
|
|
labelSelector:
|
||
|
|
matchExpressions:
|
||
|
|
- key: app
|
||
|
|
operator: In
|
||
|
|
values:
|
||
|
|
- order-service
|
||
|
|
topologyKey: kubernetes.io/hostname
|
||
|
|
# ^^^ "尽量"把同 app 的 Pod 分配到不同机器
|
||
|
|
```
|
||
|
|
|
||
|
|
> [!question] 为什么要把同一个服务的 Pod 分散？
|
||
|
|
>
|
||
|
|
> 如果所有副本都在同一台物理机上，这台机器宕机就会导致服务完全不可用。**故障域隔离**是高可用的基础——把 Pod 分散到不同节点 = 把鸡蛋放进不同篮子。
|
||
|
|
|
||
|
|
### 节点亲和性：只调度到特定标签的节点
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
spec:
|
||
|
|
affinity:
|
||
|
|
# ========== 节点亲和性：只调度到特定标签的节点 ==========
|
||
|
|
nodeAffinity:
|
||
|
|
requiredDuringSchedulingIgnoredDuringExecution:
|
||
|
|
nodeSelectorTerms:
|
||
|
|
- matchExpressions:
|
||
|
|
- key: node-role
|
||
|
|
operator: In
|
||
|
|
values:
|
||
|
|
- high-memory # 只调度到有 high-memory 标签的节点
|
||
|
|
```
|
||
|
|
|
||
|
|
### 调度级别对比
|
||
|
|
|
||
|
|
| 级别 | 含义 | 推荐度 |
|
||
|
|
|---------|------|--------|
|
||
|
|
| `requiredDuring...` | 硬约束，不满足就不调度 | 高（适用于特殊硬件节点） |
|
||
|
|
| `preferredDuring...` | 软约束，尽力而为 | 高（适用于反亲和性拆分） |
|
||
|
|
| `ignoringDuringExecution` | 调度时检查，运行时忽略变化 | 标准做法 |
|
||
|
|
|
||
|
|
## Taint & Toleration — 节点级别的"白名单"
|
||
|
|
|
||
|
|
Taint 设在 Node 上，Toleration 设在 Pod 上：
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# 给节点加污点：只有带 db 容忍度的 Pod 才能调度上来
|
||
|
|
kubectl taint nodes node1 dedicated=db:NoSchedule
|
||
|
|
```
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
spec:
|
||
|
|
tolerations:
|
||
|
|
- key: "dedicated"
|
||
|
|
operator: "Equal"
|
||
|
|
value: "db"
|
||
|
|
effect: "NoSchedule"
|
||
|
|
```
|
||
|
|
|
||
|
|
> [!info] Taint effect 三种类型
|
||
|
|
>
|
||
|
|
> | Effect | 含义 | 场景 |
|
||
|
|
> |--------|------|------|
|
||
|
|
> | `NoSchedule` | 不接受新 Pod | 专用节点 |
|
||
|
|
> | `PreferNoSchedule` | 尽量避免但不强制 | 优先但不是硬性 |
|
||
|
|
> | `NoExecute` | 驱逐已有的 Pod | 节点出问题时 |
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
flowchart LR
|
||
|
|
Node["Node 打 Taint<br/>dedicated=db:NoSchedule"] --> Check{Pod 有 Toleration?}
|
||
|
|
|
||
|
|
Check -- 否 --> Reject["❌ 不允许调度"]
|
||
|
|
Check -- 是 --> Accept["✅ 允许调度"]
|
||
|
|
|
||
|
|
style Reject fill:#ffebee
|
||
|
|
style Accept fill:#e8f5e9
|
||
|
|
```
|
||
|
|
|
||
|
|
## TopologySpreadConstraints — K8s 1.19+ 推荐的跨可用区分布
|
||
|
|
|
||
|
|
比 anti-affinity 更优雅的实现方式，特别适合多云和混合云架构：
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
spec:
|
||
|
|
topologySpreadConstraints:
|
||
|
|
- maxSkew: 1 # 任何两个可用区最多差 1 个 Pod
|
||
|
|
topologyKey: topology.kubernetes.io/zone # 按可用区分组
|
||
|
|
whenUnsatisfiable: DoNotSchedule # 不满足则不调度
|
||
|
|
labelSelector:
|
||
|
|
matchLabels:
|
||
|
|
app: order-service
|
||
|
|
```
|
||
|
|
|
||
|
|
> [!tip] WhenUnsatisfiable 策略选择
|
||
|
|
>
|
||
|
|
> - `DoNotSchedule`：宁可等也不破坏分布均衡（严格一致）
|
||
|
|
> - `ScheduleAnyway`：先调度上去，能分尽量分（可用性优先）
|
||
|
|
>
|
||
|
|
> **生产建议**：对外服务用 `ScheduleAnyway` + 反亲和性；内部有状态组件用 `DoNotSchedule`。
|
||
|
|
|
||
|
|
## 补充：NodeSelector 最简单的限制
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
spec:
|
||
|
|
nodeSelector:
|
||
|
|
disktype: ssd # 只调度到标注了 disktype=ssd 的节点
|
||
|
|
```
|
||
|
|
|
||
|
|
> [!note] NodeSelector vs Node Affinity
|
||
|
|
>
|
||
|
|
> NodeSelector 是最基本的调度方式——只能做相等匹配。如果需要 `In/NotIn/Exists` 等更复杂的逻辑，应使用 `nodeAffinity`。现代 K8s 项目中建议统一使用 `nodeAffinity` 以保持风格一致。
|
||
|
|
|
||
|
|
## 组合使用示例
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
# 一个典型的 Production Web 部署
|
||
|
|
spec:
|
||
|
|
template:
|
||
|
|
spec:
|
||
|
|
# ① 必须运行在 Linux x86_64 节点上
|
||
|
|
affinity:
|
||
|
|
nodeAffinity:
|
||
|
|
requiredDuringSchedulingIgnoredDuringExecution:
|
||
|
|
nodeSelectorTerms:
|
||
|
|
- matchExpressions:
|
||
|
|
- key: kubernetes.io/os
|
||
|
|
operator: In
|
||
|
|
values: [linux]
|
||
|
|
- key: kubernetes.io/arch
|
||
|
|
operator: In
|
||
|
|
values: [amd64]
|
||
|
|
|
||
|
|
# ② 尽量分散在不同可用区
|
||
|
|
topologySpreadConstraints:
|
||
|
|
- maxSkew: 1
|
||
|
|
topologyKey: topology.kubernetes.io/zone
|
||
|
|
whenUnsatisfiable: ScheduleAnyway
|
||
|
|
labelSelector:
|
||
|
|
matchLabels:
|
||
|
|
app: web-frontend
|
||
|
|
|
||
|
|
# ③ 避免与后台批处理任务混在同一节点
|
||
|
|
podAntiAffinity:
|
||
|
|
preferredDuringSchedulingIgnoredDuringExecution:
|
||
|
|
- weight: 200
|
||
|
|
podAffinityTerm:
|
||
|
|
labelSelector:
|
||
|
|
matchExpressions:
|
||
|
|
- key: workload-type
|
||
|
|
operator: In
|
||
|
|
values: [batch-job]
|
||
|
|
topologyKey: kubernetes.io/hostname
|
||
|
|
```
|
||
|
|
|
||
|
|
## 关联笔记
|
||
|
|
|
||
|
|
- [[../01-核心概念与Deployment]] — Deployment 中的 podTemplateSpec 位置
|
||
|
|
- [[../06-资源与安全管控]] — ResourceQuota / NetworkPolicy 配合调度实现安全隔离
|
||
|
|
- [[../hhs/MS/05-部署运维/02-Kubernetes]] — 调度失控时的常见排查技巧
|