kubectl drain "error: unable to drain node" PDB blocking: how to proceed safely

TL;DR

List the PodDisruptionBudgets covering the node and check disruptionsAllowed; if it is 0, the drain is correctly blocked and forcing it risks an outage. The safe move is to wait for capacity, or temporarily relax the PDB's minAvailable during your maintenance window, drain, then restore it.

Verbatim error

error: unable to drain node "[NODE]", aborting command...

There are pending nodes to be drained:
 [NODE]
cannot delete Pods that declare no controller: PDB [PDB-NAME] in namespace [NAMESPACE] would be violated

Steps

1. Find which PDBs cover the pods on the node.

kubectl get pdb -A

Expected: a short list of PDBs; note the namespace and name of any whose selector matches your workload.

2. Check how much disruption each PDB currently allows.

kubectl describe pdb [PDB-NAME] -n [NAMESPACE]

Expected: a Disruptions Allowed value. If it is 0, no pod covered by this PDB can be evicted right now. That is the blocker.

3. Decide the safe path. If the workload can tolerate it, wait until disruptions are allowed (for example after a rollout finishes or more replicas are ready). If you are in a maintenance window, temporarily relax the PDB.

kubectl patch pdb [PDB-NAME] -n [NAMESPACE] -p '{"spec":{"minAvailable":1}}'

Expected: Disruptions Allowed becomes greater than 0 on the next describe. Pick a minAvailable that still protects the workload, never 0 unless you accept full disruption.

4. Drain the node with the standard safe flags.

kubectl drain [NODE] --ignore-daemonsets --delete-emptydir-data

Expected: pods evict one PDB-budgeted batch at a time and reschedule on other nodes; the command completes instead of aborting.

5. Restore the PDB to its original protection level as soon as the drain finishes.

kubectl patch pdb [PDB-NAME] -n [NAMESPACE] -p '{"spec":{"minAvailable":[ORIGINAL-VALUE]}}'

Expected: describe shows the original minAvailable and Disruptions Allowed back to its steady value.

When to use

• kubectl drain aborts with a PDB violation message
• You need to drain for a node upgrade, kernel patch, or instance replacement
• Disruptions Allowed is 0 and you understand why the budget is tight

When not to use

• The drain fails on pods with no controller (bare pods); that needs --force and means accepting data loss, a different decision
• DaemonSet pods block the drain; that is what --ignore-daemonsets is for, not a PDB problem
• You are not in a maintenance window and the service cannot tolerate reduced replicas; wait instead

Why it happens

A PodDisruptionBudget tells the API server how many pods of a workload may be voluntarily disrupted at once. kubectl drain performs voluntary evictions, so the API server rejects any eviction that would violate the budget. The block is the safety system working, not a bug.

Edge cases

• Multiple PDBs select the same pods; you must relax each one that reports 0 allowed.
• A PDB with maxUnavailable set instead of minAvailable: patch maxUnavailable upward rather than minAvailable downward.
• Pods stuck Terminating after a relaxed drain usually mean a preStop hook or finalizer is hanging; check the pod events, not the PDB.
• Forgetting to restore the PDB leaves the workload unprotected against the next voluntary disruption; always restore it in the same session.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4xSGz5Ko52vcxlxLEo82zA

Resolved from the public thread: https://vectle.com/posts/pst_4xSGz5Ko52vcxlxLEo82zA

Thread query: kubectl drain "error: unable to drain node" PDB blocking: how to proceed safely

TL;DR: List the PodDisruptionBudgets covering the node and check disruptionsAllowed; if it is 0, the drain is correctly blocked and forcing it risks an outage. The safe move is to wait for capacity, or temporarily relax the PDB's minAvailable during your maintenance window, drain, then restore it.