Your Node Drain Hung Because of a Pod Disruption Budget

The cluster upgrade was routine until it was not. The autoscaler drained three nodes fine and hung on the fourth, where a deployment lived whose Pod Disruption Budget said minAvailable: 3 and which had exactly three replicas. The drain asked to evict one pod. The PDB said no, because evicting one would leave two, below the minimum. So the drain waited for a disruption allowance that could never arrive, and the upgrade sat for hours behind a safety mechanism doing precisely what it was configured to do.

PDBs are one of the better ideas in Kubernetes, and one of the most common sources of a stuck cluster operation, because the configuration that feels safest at write time is the configuration that blocks all voluntary disruption at drain time.

This was Kubernetes 1.32. The eviction and PDB semantics are stable.

What a PDB actually protects

Kubernetes distinguishes voluntary and involuntary disruption. A node drain, a cluster autoscaler scale down, a control plane initiated upgrade are voluntary, and they respect PDBs. A hardware failure, a kernel panic, a cloud provider losing the host are involuntary, and they do not. A PDB therefore does not make your workload durable. It makes your maintenance operations polite.

The budget says how many pods may be down at once during voluntary disruption, either as minAvailable or maxUnavailable. During a drain, the eviction API checks the budget, and if evicting would violate it, the eviction is refused and the drainer waits and retries.

The configuration that hangs everything

The hang in the incident is the classic shape: minAvailable set to the full replica count. With three replicas and minAvailable three, the allowed disruptions are zero, forever. No voluntary eviction can ever proceed, so every drain, every autoscaler scale down, every upgrade that needs to move those pods will stall on them.

The author's intent was "never be down", which is a fine wish and an impossible contract. Maintenance requires at least one pod to move at a time, and a budget that allows zero movement converts every maintenance operation into a manual intervention.

The equally sneaky variant is a percentage that rounds to the full count. minAvailable: 100% is the same trap with different spelling.

The other failure mode: a budget that protects nothing

The opposite misconfiguration is a PDB that is trivially satisfiable, minAvailable: 1 on a nine replica service, which permits eight simultaneous evictions. During a zone level maintenance event the autoscaler can then evict most of the service at once, and the PDB, present and green, has protected almost nothing.

So the budget must be set from the question "how many of these can be down at once while the service still works", which is a capacity and latency question, not a feeling. If the service needs two of three to hold load, minAvailable two is the honest number, and it permits exactly one disruption at a time, which is what the drain needs to make progress slowly and safely.

Seeing why a drain is stuck

The drain's own output says it is waiting on a PDB, but the clearer view is the budget's status:

kubectl get pdb
kubectl describe pdb my-service-pdb

The allowed disruptions field is the number you want. Zero means every voluntary eviction is currently refused, and if a drain is in flight, that is your hang.

Then confirm the arithmetic: replicas versus minAvailable. When they are equal, you have found the incident and the fix in the same glance.

Also check which pods are pending eviction, because a drain can also wait on pods that have no PDB but cannot terminate, a stuck finalizer or a missing preStop handling, and it is worth separating the PDB wait from the termination wait. The termination side is the graceful shutdown story in the thirty seconds between SIGTERM and SIGKILL.

The fix and the shape that works

For the stuck upgrade, the immediate fix is to correct the budget to allow one disruption, then let the drain proceed. Do not delete the PDB to unblock, because deleting removes the protection you wanted exactly during the operation that most needs it.

The working shape, for a service with N replicas that needs K to hold load, is minAvailable: K where K is less than N, or equivalently maxUnavailable: N-K. The autoscaler and the drain then move pods one at a time, waiting for replacements to become ready between evictions, which is slow and correct.

For single replica services the PDB cannot help at all, because any eviction violates any budget. The honest fix there is not a budget but making the workload tolerable to lose, or accepting that its node operations need manual handling.

If it happens again

A Pod Disruption Budget is a contract between your service's availability and your cluster's ability to maintain itself, and the contract must permit at least one voluntary disruption or maintenance will stall on it forever. Set the budget from the number of replicas the service genuinely needs, verify allowed disruptions is at least one under normal conditions, and treat a zero as both a hang waiting to happen and a sign the number was a wish rather than a measurement.

The eviction priority that decides who dies when the budget cannot save everyone is the QoS story in requests, limits and the QoS class.

The rollout interaction worth knowing

A PDB also interacts with your own deployments, and the interaction surprises people. A rolling update that evicts and replaces pods is a voluntary disruption, so a tight budget slows the rollout to one pod at a time with readiness gates between, which is usually what you want. But a budget combined with a slow readiness probe can make a rollout take so long that operators raise the budget to speed it up, and raising it to speed a deploy is exactly the moment the protection stops meaning what it said. If rollouts are too slow, fix the readiness signal, not the budget, because the budget is the number that gets read during the next zone failure, not during the deploy.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud