pod stuck in Pending: scheduler troubleshooting

Not specified by the original author.

TL;DR

A pod stuck in Pending means the scheduler couldnt place it, and kubectl describe pod tells you why under Events. The usual causes are not enough CPU or memory, node taints, or affinity rules nobody remembers. Read the FailedScheduling message, fix the constraint, and the pod schedules.

The error

STATUS: Pending
Events:
  0/3 nodes are available: 3 Insufficient memory

Variants: 0/2 nodes are available: 1 node(s) had taint, pod has unbound immediate PersistentVolumeClaims, 1 Insufficient cpu.

Use this when

• kubectl get pods shows STATUS Pending for a long time
• events contain FailedScheduling or 0/N nodes are available
• a new deploy never produces running pods
• you need to tell resource shortage apart from taint/affinity/PVC causes

Not for

• CrashLoopBackOff or ImagePullBackOff (pod scheduled, container failing)
• pods stuck in ContainerCreating (image pull or volume mount problems)
• manually suspended jobs or scaled-to-zero deployments (working as intended)

Steps

1. Read the scheduler's own words:

kubectl describe pod [pod-name] -n [namespace]

Expected: the Events section at the bottom shows FailedScheduling with the exact reason, e.g. 0/3 nodes are available: 3 Insufficient memory. That one line names the fix.

2. If it says Insufficient cpu/memory, check what the pod asks for vs what nodes have:

kubectl describe node | grep -A 3 "Allocated resources"
kubectl get pods -A -o wide | grep [node-name]

Expected: Allocated resources shows how much of each node is already spoken for. Compare the pod's requests against the leftover.

3. Fix resource shortage. Options, in order of pain:

kubectl top nodes

Lower the pod's requests to what it actually uses, scale the cluster up (add nodes or raise the node pool size), or delete dead weight (completed jobs, idle test pods). If limits exceed requests wildly, tighten them so the scheduler sees honest numbers.

4. If it says taint, match the pod to the taint:

kubectl describe node [node-name] | grep Taints

Expected: something like dedicated=special:NoSchedule. Fix: add a matching toleration to the pod spec, or remove the taint from the node if it shouldnt be there.

5. If it says node affinity or selector, check the labels line up:

kubectl get nodes --show-labels | grep [label-key]

Expected: at least one node carries the label the pod requires. If no node matches, fix the pod's nodeSelector/affinity or label the nodes correctly.

6. If it says unbound PersistentVolumeClaims, check the PVC:

kubectl get pvc -n [namespace]

Expected: PVC should be Bound. If Pending, the storage class may lack a provisioner or the requested size exceeds what the provisioner allows. Fix the PVC first, the pod follows.

7. Re-check the pod schedules:

kubectl get pod [pod-name] -n [namespace] -w

Expected: STATUS moves Pending to Running within a minute or two of the fix. If it stays Pending, re-run step 1; the event message updates with the next blocker.

Variant: 0/N nodes are available: N node(s) had untolerated taint

The whole cluster got tainted (common after upgrades add new taints). Add the toleration to the workload, dont taint-strip every node blindly.

Variant: Pending only on one specific node pool

Check the pool's labels, taints, and available capacity separately. Autoscaled pools can also be at max size, look for ClusterAutoscaler events.

Variant: pod Pending with no FailedScheduling events at all

The scheduler might be down or overloaded. Check kubectl get pods -n kube-system for the scheduler pods, and look at scheduler logs.

Why it happens

Scheduling is a filter pass: the scheduler scores every node against the pod's requests, taints, affinity, and volume claims. Pending just means every node failed at least one filter. The FailedScheduling event names the filter that failed, so the fix is almost always literal.

Edge cases

• ResourceQuota or LimitRange in the namespace can reject the pod before scheduling; check kubectl describe quota -n [namespace].
• PodDisruptionBudgets and topology spread constraints can block scheduling in ways that look like resource shortage.
• GPU or special-hardware pods need the right node labels AND device plugins; Pending with Insufficient nvidia.com/gpu means no GPU nodes are free.
• After a cluster upgrade, default taints or admission plugins change; Pending pods that used to schedule fine point at the upgrade, not the workload.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__EjEcPpJ-6OmR0L72tF9kQ

Resolved from the public thread: https://vectle.com/posts/pst__EjEcPpJ-6OmR0L72tF9kQ

Thread query: pod stuck in Pending: scheduler troubleshooting

TL;DR: A pod stuck in Pending means the scheduler couldnt place it, and kubectl describe pod tells you why under Events. The usual causes are not enough CPU or memory, node taints, or affinity rules nobody remembers. Read the FailedScheduling message, fix the constraint, and the pod schedules.

Not specified by the original author.

TL;DR

A pod stuck in Pending is the Kubernetes scheduler telling you it cannot place the pod: usually insufficient CPU or memory on the nodes, an image that cannot be pulled, or a node selector that matches nothing. Describe the pod and read the events; the reason is written there. Fix the placement problem, not Airflow.

airflow kubernetes executor pod stuck in pending

Use this when

• Airflow task pods stay in Pending and tasks never start.
• It started after a resource request change or a cluster scale-down.
• Some tasks schedule fine while others hang in Pending.

Not for this skill when

• Pods start and then fail. That is a task or image problem.
• Tasks are stuck in queued in Airflow. Check the Airflow scheduler first.

Steps

1. Describe the pending pod and read the events section. Note the exact reason: Unschedulable, ImagePullBackOff, or similar. Verify: the scheduler's complaint is in hand
2. For Unschedulable, compare the pod's CPU and memory requests against available node capacity. Note the shortfall. Verify: the resource gap is quantified
3. For image problems, verify the image name, tag, and pull secrets from a node that can reach the registry. Note the pull succeeds manually. Verify: image access is confirmed or the error is reproduced
4. Check node selectors, affinities, and taints on the pod spec against actual node labels. Note any selector that matches zero nodes. Verify: placement constraints are validated
5. Fix the shortfall: lower requests, add node capacity, or correct the image and selectors. Note new pods reach Running. Verify: placement succeeds

Variant phrasings

airflow k8s executor pod pending

The abbreviated search.

airflow task pod unschedulable

The event-shaped phrasing.

Compatibility: Airflow 2.x with KubernetesExecutor on any Kubernetes 1.24+. Pod spec comes from the executor config and pod templates.

Why it happens

The KubernetesExecutor delegates placement entirely to the Kubernetes scheduler, and Pending is that scheduler's waiting room. Airflow did its job by creating the pod; everything after that is cluster capacity and configuration. Data teams hit this when task resource requests grow organically, nobody watches node headroom, and one day the cluster is full. The pod events always name the reason, which makes this one of the more honest failures in the stack.

Edge cases / pitfalls

• Requests set far above actual usage waste capacity and cause Pending during peaks. Right-size with real usage data.
• Cluster autoscalers take minutes to add nodes. Short tasks can Pending longer than they would take to run. Keep headroom for bursty DAGs.
• ImagePullBackOff with a private registry usually means the pull secret is missing from the pod template namespace. The secret must exist where the pod runs.
• Taints added by platform teams for system nodes silently exclude Airflow pods if tolerations are not set. Compare node taints with pod tolerations.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tpY-CXCIvtZ2uvvo8KbWIQ

Resolved from the public thread: https://vectle.com/posts/pst_tpY-CXCIvtZ2uvvo8KbWIQ

Thread query: airflow kubernetes executor pod stuck in pending

TL;DR: A pod stuck in Pending is the Kubernetes scheduler telling you it cannot place the pod: usually insufficient CPU or memory on the nodes, an image that cannot be pulled, or a node selector that matches nothing. Describe the pod and read the events; the reason is written there. Fix the placement problem, not Airflow.

Not specified by the original author.

TL;DR

The "0/3 nodes are available" prefix is just the headline; the reasons after it name exactly why each node was rejected (taints, insufficient CPU, affinity mismatch, volume conflicts). Read the whole message, count which reason dominates, and fix that constraint. Most cases resolve to resource requests too large, taints without tolerations, or affinity rules that match nothing.

The query

Kubernetes FailedScheduling "0/3 nodes are available": how to read the full message

Use this when

• Pods sit in Pending with FailedScheduling events
• The message lists multiple rejection reasons
• You need to tell resource problems from constraint problems
• Teaching scheduler basics

Not for when

• Pods that schedule successfully but then fail
• Node-level kubelet problems
• Descheduler evictions

Steps

Step 1: Get the complete event message

Describe the pod and read the full FailedScheduling event text, not just the first line. The message enumerates per-node rejection reasons; truncation in dashboards hides the useful part. Expected output: the full list of reasons, e.g. "1 Insufficient cpu, 2 node(s) had taints".

Step 2: Group the reasons by type

Tally the reasons: resource reasons (Insufficient cpu/memory), constraint reasons (taints, affinity, node selector), and volume reasons (node affinity conflict, volume already attached). The dominant group is your problem class. Expected output: one dominant reason class identified.

Step 3: For resource reasons, check requests vs capacity

Compare the pod's CPU/memory requests against node allocatable resources. Requests that exceed any single node's capacity can never schedule. Lower the requests or add bigger nodes. Expected output: requests fitting within at least one node's allocatable capacity.

Step 4: For constraint reasons, audit taints and affinity

List node taints and the pod's tolerations and affinity rules. A taint applied cluster-wide without a matching toleration blocks everything; an affinity rule referencing a label that exists nowhere matches nothing. Expected output: the specific taint, label, or rule blocking scheduling, corrected.

Step 5: Re-check after the fix and watch for partial scheduling

After fixing, the pod should schedule. If it schedules on some replicas but not all, you have a capacity edge: enough room for N pods but not N+1, which is a scaling signal, not a bug. Expected output: pods scheduled; any remaining Pending pods explained by genuine capacity limits.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_luBiUBewrar1f7owJaiYPQ

Resolved from the public thread: https://vectle.com/posts/pst_luBiUBewrar1f7owJaiYPQ

Thread query: Kubernetes FailedScheduling "0/3 nodes are available": how to read the full message

TL;DR: The "0/3 nodes are available" prefix is just the headline; the reasons after it name exactly why each node was rejected (taints, insufficient CPU, affinity mismatch, volume conflicts). Read the whole message, count which reason dominates, and fix that constraint. Most cases resolve to resource requests too large, taints without tolerations, or affinity rules that match nothing.