Stop Breaking Prod: PodDisruptionBudget and topologySpreadConstraints Done Right

A cluster upgrade is supposed to be boring. You click "upgrade", the control plane rolls, then the worker node pool rolls node by node: cordon, drain, reboot, uncordon. Twenty minutes later someone pings you in Slack: "checkout was down for 90 seconds". Nobody deployed anything. Nobody touched the app.

What happened is simple: the drain evicted all the replicas of your service at once, or it evicted the only replica that was serving traffic, because nothing told Kubernetes that those pods were related and that killing them together was unacceptable.

Two objects fix this: PodDisruptionBudget (PDB), which caps how many pods of a group can be voluntarily disrupted at the same time, and topologySpreadConstraints, which makes sure those pods were never all on the same node or in the same zone to begin with. They only work together. A PDB without spreading just makes the drain hang; spreading without a PDB just makes the outage shorter.

This article walks through writing both correctly, and verifying them with kubectl drain --dry-run=server before the upgrade window, not during it.

Context: what a node drain actually does

Kubernetes distinguishes two kinds of pod disruptions.

Involuntary disruptions are hardware failure, kernel panic, OOM kill, a spot instance reclaimed. Nothing protects you here except having replicas in different failure domains.

Voluntary disruptions are the ones you (or the platform) trigger: kubectl drain, a node pool upgrade, a MachineConfig rollout in OpenShift, cluster autoscaler scale-down, a node replacement by Karpenter or the Machine API. These go through the Eviction API (pods/eviction subresource), and the Eviction API is the only thing that honors a PDB.

The flow for kubectl drain node-1:

1. Cordon the node (spec.unschedulable: true) so no new pod lands there.
2. List pods on the node, skip DaemonSet pods and mirror pods.
3. For each remaining pod, POST an Eviction object.
4. The API server checks every PDB whose selector matches the pod. If evicting it would push the group below the budget, it returns HTTP 429 Too Many Requests, and kubectl retries until the budget frees up or the timeout expires.

Key consequence for beginners: if no PDB matches your pods, every eviction is accepted immediately. Drain will happily delete all three replicas of your API in the same second if they live on the same node.

And the second consequence: kubectl delete pod and a direct node shutdown bypass PDBs entirely. A PDB is not a guarantee against everything; it is a guarantee against the eviction path.

Writing a PodDisruptionBudget that actually holds

Here is a complete, working example for a stateless HTTP service with 3 replicas.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
  namespace: shop
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: checkout-api
      app.kubernetes.io/instance: checkout-api
  # Kubernetes >= 1.27 / OpenShift >= 4.14
  unhealthyPodEvictionPolicy: AlwaysAllow

Three things to understand in that manifest.

minAvailable or maxUnavailable?

Both are valid, you pick one, and they behave very differently when the replica count changes.

• maxUnavailable: 1 — "at most one pod of this group may be down at a time". This is the right default for anything horizontally scalable. It keeps working if you scale from 3 to 10 replicas.
• minAvailable: 2 — "always keep at least 2 ready pods". Useful when you have a hard quorum requirement (etcd-like, 3-node clusters, leader election).

Percentages are allowed (maxUnavailable: 25%) and are rounded in your favour, but on small deployments percentages behave surprisingly: with 3 replicas, minAvailable: 50% rounds up to 2. Be explicit with small numbers.

Never write minAvailable: 100% or maxUnavailable: 0. That is not high availability, that is a permanent block on node maintenance, and a cluster admin will eventually drain the node with --disable-eviction, which deletes your pods without any budget at all.

The selector must match exactly one logical group

A PDB is not attached to a Deployment. It is attached to a label selector. If you write selector.matchLabels: {app: api} and three different Deployments in the namespace use app: api, the budget is shared across all of them and the drain math is wrong.

Verify the selector before trusting it:

# What the PDB actually selects
kubectl get pods -n shop \
  -l app.kubernetes.io/name=checkout-api,app.kubernetes.io/instance=checkout-api \
  -o wide

Also: a PDB only protects pods in its own namespace. Copy-pasting a PDB without changing the namespace silently protects nothing. Since Kubernetes 1.21 the API will reject two PDBs matching the same pod for eviction purposes with an error, so keep one PDB per workload.

unhealthyPodEvictionPolicy: the trap that hangs your upgrade

Classic incident: a bad image is rolled out, pods are CrashLoopBackOff, so zero pods are Ready. The PDB says "at most 1 unavailable", and since all of them are already unavailable, every eviction is rejected. The node drain blocks forever on pods that were broken anyway.

unhealthyPodEvictionPolicy: AlwaysAllow fixes this: pods that are running but not Ready can always be evicted, regardless of the budget. The default (IfHealthyBudget) only allows that when the budget is currently satisfied. Set AlwaysAllow for any workload where an unhealthy pod serves no traffic — which is almost all of them behind a Service.

The single-replica case

spec:
  replicas: 1

with

spec:
  minAvailable: 1

means the pod can never be evicted. Drain hangs. Node pool upgrade stalls. You have two honest options: run 2+ replicas, or accept the disruption and use maxUnavailable: 1 (which for a single replica means "yes, you may evict it"). Do not pretend a single replica is highly available.

Spreading pods so the budget can be respected

A PDB only limits the rate of disruption. If all 3 replicas are on node-1, draining node-1 means three evictions, one at a time, each waiting for the previous pod to be rescheduled and Ready on another node. Best case this is slow; worst case the replacement pods are Pending (no capacity, cordoned nodes during an upgrade) and the drain blocks.

topologySpreadConstraints prevents the situation entirely by spreading pods across failure domains at scheduling time.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: shop
  labels:
    app.kubernetes.io/name: checkout-api
    app.kubernetes.io/instance: checkout-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: checkout-api
      app.kubernetes.io/instance: checkout-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: checkout-api
        app.kubernetes.io/instance: checkout-api
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: checkout-api
              app.kubernetes.io/instance: checkout-api
          matchLabelKeys:
            - pod-template-hash
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: checkout-api
              app.kubernetes.io/instance: checkout-api
      containers:
        - name: api
          image: registry.example.com/shop/checkout-api:1.8.3
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 5
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "10"]
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi
      terminationGracePeriodSeconds: 45

Reading the constraints

• maxSkew: 1 on topology.kubernetes.io/zone with DoNotSchedule: the difference in pod count between the busiest and the emptiest zone may not exceed 1. With 3 replicas and 3 zones, you get exactly 1 per zone. A whole zone can go down (or be upgraded) and you keep 2/3 capacity.
• maxSkew: 1 on kubernetes.io/hostname with ScheduleAnyway: prefer one pod per node, but do not block scheduling if the cluster is tight. Using DoNotSchedule on hostname is a common way to end up with Pending pods during an upgrade, when half the nodes are cordoned. Start soft.
• matchLabelKeys: [pod-template-hash] (stable since Kubernetes 1.29, beta in 1.27) scopes the spreading calculation to the current ReplicaSet. Without it, old pods from the previous rollout count toward the skew and rolling updates get stuck. If your cluster is older, use ScheduleAnyway on the zone constraint instead.

Rollout strategy matters too

maxUnavailable: 0 + maxSurge: 1 means a rolling update never reduces capacity below replicas. Combined with maxUnavailable: 1 in the PDB, you have a consistent story: deployments never reduce availability, drains reduce it by at most one pod.

Also note preStop: sleep 10 and terminationGracePeriodSeconds: 45. When a pod is evicted, it is removed from Service endpoints asynchronously. The sleep gives kube-proxy, the ingress controller and the service mesh time to stop sending traffic before the process exits. Without it, a perfectly budgeted drain still produces a handful of 502s.

Verifying before the upgrade window

1. Check what the PDB currently allows

kubectl get pdb -n shop

NAME           MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
checkout-api   N/A             1                 1                     12d

ALLOWED DISRUPTIONS is the number that matters. If it is 0 while everything looks fine, your budget is too tight or pods are not Ready. If the EXPECTED PODS count in kubectl get pdb -o yaml is 0, your selector matches nothing — the PDB is decorative.

Cluster-wide scan for budgets that will block maintenance:

kubectl get pdb -A -o json | jq -r '
  .items[]
  | select(.status.disruptionsAllowed == 0)
  | "\(.metadata.namespace)/\(.metadata.name) expected=\(.status.expectedPods) desired=\(.status.desiredHealthy)"'

And find workloads with no PDB at all:

kubectl get deploy -A -o json | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name)"' | sort > /tmp/deploys
kubectl get pdb -A -o json | jq -r '.items[] | "\(.metadata.namespace)"' | sort -u > /tmp/ns-with-pdb
comm -23 /tmp/deploys <(grep -f /tmp/ns-with-pdb /tmp/deploys 2>/dev/null | sort)

2. Dry-run the drain

--dry-run=server sends real Eviction requests with dryRun: ["All"]. The API server runs the full PDB admission check and then throws the deletion away. This is as close as you get to a rehearsal without touching production.

kubectl drain worker-02.example.com \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --dry-run=server

On OpenShift, the same command with oc adm drain:

oc adm drain worker-02.example.com \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --dry-run=server

If a PDB would reject the eviction, you get an explicit error instead of an outage:

error when evicting pods/"checkout-api-6d4f8c9b7-x2k9p" -n "shop":
Cannot evict pod as it would violate the pod's disruption budget.

Two limitations to keep in mind. Server-side dry-run evaluates each eviction against the current state — it does not simulate the cascade of N sequential evictions, and it does not verify that the rescheduled pods will find room elsewhere. Use --dry-run=client first if you only want the list of affected pods.

3. Do a real drain on one node, during business hours

# Rehearse on a single node, with a timeout so it cannot hang forever
kubectl drain worker-02.example.com \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=300s

# Watch what moves
kubectl get pods -n shop -o wide -w

# Put the node back
kubectl uncordon worker-02.example.com

If that drain finishes in under five minutes with no 5xx in your dashboards, the node pool upgrade will behave the same way, node after node.

4. OpenShift: check the MachineConfigPool pace

On OpenShift, worker upgrades are driven by the MachineConfigPool, which drains one node at a time by default:

oc get machineconfigpool
oc get mcp worker -o jsonpath='{.spec.maxUnavailable}{"\n"}'

Keep maxUnavailable: 1 on the worker pool unless you have verified that every workload tolerates two simultaneous node drains. The built-in PodDisruptionBudgetAtLimit and PodDisruptionBudgetLimit alerts in the cluster monitoring stack will fire before the upgrade if a budget is already saturated — treat them as upgrade blockers, not noise.

Security best practices

PDBs sit at the intersection of availability and operations, which makes them an interesting target and an easy foot-gun.

• Treat a PDB as a denial-of-maintenance primitive. Any user who can create a PDB in a namespace can block node drains cluster-wide (a drain stops on the first blocked pod). Restrict policy/v1 PodDisruptionBudget write access with RBAC, and enforce a sane shape with a ValidatingAdmissionPolicy, Kyverno or OPA Gatekeeper: reject maxUnavailable: 0, reject minAvailable equal to or greater than replicas, require a maxUnavailable on deployments with more than one replica.
• Lock down the eviction subresource. create on pods/eviction is what drain uses. Standard developer roles do not need it; cluster maintenance roles do. Audit it: kubectl auth can-i create pods/eviction --as=system:serviceaccount:dev:builder -n shop.
• Never normalize --disable-eviction or --force. Both bypass PDBs (--disable-eviction uses raw deletes; --force removes unmanaged pods). If they appear in a runbook, the real problem is an over-tight budget somewhere. Log every use.
• Audit who drains what. Node drains are a high-privilege operation. Keep nodes patch/update and pods/eviction create in a dedicated ClusterRole bound to the platform team, and make sure the audit policy records pods/eviction at RequestResponse level.
• Do not let PDBs hide unhealthy workloads. unhealthyPodEvictionPolicy: AlwaysAllow is correct, but pair it with real readiness probes. A pod that reports Ready while it cannot serve traffic defeats both the PDB and the Service.
• Use PriorityClass deliberately. A high-priority pod can preempt your well-spread replicas and undo the topology layout. Reserve system-cluster-critical for actual platform components, and give application workloads an explicit, lower priorityClassName.
• Keep security context tight on the workloads you are protecting. On OpenShift, the restricted-v2 SCC applies by default; do not relax it just to make a pod restart faster. runAsNonRoot: true, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true and a dropped capability set cost nothing at drain time.

Conclusion

Surviving a node pool upgrade is not luck, it is three manifests reviewed together.

• Without a PDB, the eviction API never says no. A drain can take out every replica of a service in the same second. Every Deployment with more than one replica needs a policy/v1 PodDisruptionBudget.
• maxUnavailable: 1 is the sane default for stateless services; minAvailable: N is for quorum-based systems. Never maxUnavailable: 0 or minAvailable: 100%.
• Set unhealthyPodEvictionPolicy: AlwaysAllow so a crash-looping rollout cannot block cluster maintenance.
• A PDB without topologySpreadConstraints only slows the outage down. Spread across topology.kubernetes.io/zone with DoNotSchedule, across kubernetes.io/hostname with ScheduleAnyway, and add matchLabelKeys: [pod-template-hash] so rollouts are not blocked by their own old pods.
• Add maxUnavailable: 0 / maxSurge: 1 to the rollout strategy and a preStop sleep so capacity never dips and connections drain cleanly.
• Verify before the maintenance window: kubectl get pdb -A for ALLOWED DISRUPTIONS: 0, then kubectl drain --dry-run=server, then one real drain on one node during business hours.

Ten minutes of YAML and one rehearsal drain is cheaper than one postmortem. Run the dry-run on a random worker node today — the result will tell you exactly how your next upgrade is going to go.