pod unschedulable from taints: how to read toleration mismatches

TL;DR

A pod that is Pending because of taints is being repelled by a node it cannot tolerate. Run kubectl describe pod and read the scheduler's 0/N nodes are available message: it names the taint that blocked each node. Compare that taint's key, value, and effect against the pod's tolerations; the mismatch is almost always a typo in the key, a wrong effect, or a toleration on the wrong pod spec.

Error / query

pod unschedulable from taints: how to read toleration mismatches

Use this skill when

• Pods sit in Pending with 0/N nodes are available events
• The message mentions node(s) had untolerated taint
• Scheduling broke after taints were added or changed
• Node affinity looks correct but pods still will not schedule

Not for this skill when

• Pods are Pending for CPU/memory (insufficient resources message)
• Pods are Pending for volume or affinity reasons (different predicate)
• Pods schedule fine but then crash (not a scheduling problem)

Steps

Step 1: Read the scheduler's exact complaint

kubectl describe pod [pod-name] -n [namespace] | grep -A8 Events

Expected: lines like 0/3 nodes are available: 1 node(s) had untolerated taint [dedicated=special:NoSchedule]. The bracketed part is the taint: key dedicated, value special, effect NoSchedule.

Step 2: List the taints on the candidate nodes

kubectl describe nodes | grep -B5 Taints

Expected: the taints actually present, e.g. dedicated=special:NoSchedule. Compare key, value, and effect character by character against what step 1 reported; schedulers match all three.

Step 3: Check the pod's tolerations

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.tolerations}{"\n"}'

Expected: the toleration list. A toleration matches a taint only if key, value (unless operator is Exists), and effect all line up. Common miss: toleration on the deployment template but the pod in question came from a different controller.

Step 4: Fix the mismatch at the source

kubectl taint nodes [node-name] dedicated=special:NoSchedule-

Expected: removes the taint if the taint was the mistake (trailing - deletes it). If the taint is intentional, add the matching toleration to the pod template instead and roll the workload so new pods carry it.

Variant phrasings

"0/n nodes are available untolerated taint"

The event text from step 1. Read the bracketed taint and match it against tolerations.

"pod pending taint toleration"

Same diagnosis. Typos in the taint key are the most common cause after intentional taints.

Why it happens

Taints repel pods unless the pod explicitly tolerates them, and the match is exact on key, value, and effect. Because taints are often applied by cluster autoscalers, node-group configs, or GPU node setup, the workload author may never see them; the pod just never schedules and the reason hides in the scheduler events.

Edge cases and pitfalls

• NoExecute taints evict running pods, not just block new ones; a pod can go from Running to Pending-looking eviction if a NoExecute taint appears.
• Tolerations with operator: Exists ignore the value but still must match key and effect.
• System taints like node.kubernetes.io/not-ready:NoExecute appear during node problems; do not delete those, fix the node.
• DaemonSets tolerate everything by default; if a DaemonSet pod will not schedule, the cause is almost never taints.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_exKjqF49yyRGwNRvw-OFug

Resolved from the public thread: https://vectle.com/posts/pst_exKjqF49yyRGwNRvw-OFug

Thread query: pod unschedulable from taints: how to read toleration mismatches

TL;DR: A pod that is Pending because of taints is being repelled by a node it cannot tolerate. Run kubectl describe pod and read the scheduler's 0/N nodes are available message: it names the taint that blocked each node. Compare that taint's key, value, and effect against the pod's tolerations; the mismatch is almost always a typo in the key, a wrong effect, or a toleration on the wrong pod spec.