kubernetes pod stuck pending scheduler debug

Not specified by the original author.

TL;DR

A pod stuck in Pending means the scheduler couldnt place it, and kubectl describe pod tells you why under Events. The usual causes are not enough CPU or memory, node taints, or affinity rules nobody remembers. Read the FailedScheduling message, fix the constraint, and the pod schedules.

The error

STATUS: Pending
Events:
  0/3 nodes are available: 3 Insufficient memory

Variants: 0/2 nodes are available: 1 node(s) had taint, pod has unbound immediate PersistentVolumeClaims, 1 Insufficient cpu.

Use this when

• kubectl get pods shows STATUS Pending for a long time
• events contain FailedScheduling or 0/N nodes are available
• a new deploy never produces running pods
• you need to tell resource shortage apart from taint/affinity/PVC causes

Not for

• CrashLoopBackOff or ImagePullBackOff (pod scheduled, container failing)
• pods stuck in ContainerCreating (image pull or volume mount problems)
• manually suspended jobs or scaled-to-zero deployments (working as intended)

Steps

1. Read the scheduler's own words:

kubectl describe pod [pod-name] -n [namespace]

Expected: the Events section at the bottom shows FailedScheduling with the exact reason, e.g. 0/3 nodes are available: 3 Insufficient memory. That one line names the fix.

2. If it says Insufficient cpu/memory, check what the pod asks for vs what nodes have:

kubectl describe node | grep -A 3 "Allocated resources"
kubectl get pods -A -o wide | grep [node-name]

Expected: Allocated resources shows how much of each node is already spoken for. Compare the pod's requests against the leftover.

3. Fix resource shortage. Options, in order of pain:

kubectl top nodes

Lower the pod's requests to what it actually uses, scale the cluster up (add nodes or raise the node pool size), or delete dead weight (completed jobs, idle test pods). If limits exceed requests wildly, tighten them so the scheduler sees honest numbers.

4. If it says taint, match the pod to the taint:

kubectl describe node [node-name] | grep Taints

Expected: something like dedicated=special:NoSchedule. Fix: add a matching toleration to the pod spec, or remove the taint from the node if it shouldnt be there.

5. If it says node affinity or selector, check the labels line up:

kubectl get nodes --show-labels | grep [label-key]

Expected: at least one node carries the label the pod requires. If no node matches, fix the pod's nodeSelector/affinity or label the nodes correctly.

6. If it says unbound PersistentVolumeClaims, check the PVC:

kubectl get pvc -n [namespace]

Expected: PVC should be Bound. If Pending, the storage class may lack a provisioner or the requested size exceeds what the provisioner allows. Fix the PVC first, the pod follows.

7. Re-check the pod schedules:

kubectl get pod [pod-name] -n [namespace] -w

Expected: STATUS moves Pending to Running within a minute or two of the fix. If it stays Pending, re-run step 1; the event message updates with the next blocker.

Variant: 0/N nodes are available: N node(s) had untolerated taint

The whole cluster got tainted (common after upgrades add new taints). Add the toleration to the workload, dont taint-strip every node blindly.

Variant: Pending only on one specific node pool

Check the pool's labels, taints, and available capacity separately. Autoscaled pools can also be at max size, look for ClusterAutoscaler events.

Variant: pod Pending with no FailedScheduling events at all

The scheduler might be down or overloaded. Check kubectl get pods -n kube-system for the scheduler pods, and look at scheduler logs.

Why it happens

Scheduling is a filter pass: the scheduler scores every node against the pod's requests, taints, affinity, and volume claims. Pending just means every node failed at least one filter. The FailedScheduling event names the filter that failed, so the fix is almost always literal.

Edge cases

• ResourceQuota or LimitRange in the namespace can reject the pod before scheduling; check kubectl describe quota -n [namespace].
• PodDisruptionBudgets and topology spread constraints can block scheduling in ways that look like resource shortage.
• GPU or special-hardware pods need the right node labels AND device plugins; Pending with Insufficient nvidia.com/gpu means no GPU nodes are free.
• After a cluster upgrade, default taints or admission plugins change; Pending pods that used to schedule fine point at the upgrade, not the workload.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__EjEcPpJ-6OmR0L72tF9kQ

Resolved from the public thread: https://vectle.com/posts/pst__EjEcPpJ-6OmR0L72tF9kQ

Thread query: pod stuck in Pending: scheduler troubleshooting

TL;DR: A pod stuck in Pending means the scheduler couldnt place it, and kubectl describe pod tells you why under Events. The usual causes are not enough CPU or memory, node taints, or affinity rules nobody remembers. Read the FailedScheduling message, fix the constraint, and the pod schedules.

Not specified by the original author.

TL;DR

A pod stuck in Pending is the Kubernetes scheduler telling you it cannot place the pod: usually insufficient CPU or memory on the nodes, an image that cannot be pulled, or a node selector that matches nothing. Describe the pod and read the events; the reason is written there. Fix the placement problem, not Airflow.

airflow kubernetes executor pod stuck in pending

Use this when

• Airflow task pods stay in Pending and tasks never start.
• It started after a resource request change or a cluster scale-down.
• Some tasks schedule fine while others hang in Pending.

Not for this skill when

• Pods start and then fail. That is a task or image problem.
• Tasks are stuck in queued in Airflow. Check the Airflow scheduler first.

Steps

1. Describe the pending pod and read the events section. Note the exact reason: Unschedulable, ImagePullBackOff, or similar. Verify: the scheduler's complaint is in hand
2. For Unschedulable, compare the pod's CPU and memory requests against available node capacity. Note the shortfall. Verify: the resource gap is quantified
3. For image problems, verify the image name, tag, and pull secrets from a node that can reach the registry. Note the pull succeeds manually. Verify: image access is confirmed or the error is reproduced
4. Check node selectors, affinities, and taints on the pod spec against actual node labels. Note any selector that matches zero nodes. Verify: placement constraints are validated
5. Fix the shortfall: lower requests, add node capacity, or correct the image and selectors. Note new pods reach Running. Verify: placement succeeds

Variant phrasings

airflow k8s executor pod pending

The abbreviated search.

airflow task pod unschedulable

The event-shaped phrasing.

Compatibility: Airflow 2.x with KubernetesExecutor on any Kubernetes 1.24+. Pod spec comes from the executor config and pod templates.

Why it happens

The KubernetesExecutor delegates placement entirely to the Kubernetes scheduler, and Pending is that scheduler's waiting room. Airflow did its job by creating the pod; everything after that is cluster capacity and configuration. Data teams hit this when task resource requests grow organically, nobody watches node headroom, and one day the cluster is full. The pod events always name the reason, which makes this one of the more honest failures in the stack.

Edge cases / pitfalls

• Requests set far above actual usage waste capacity and cause Pending during peaks. Right-size with real usage data.
• Cluster autoscalers take minutes to add nodes. Short tasks can Pending longer than they would take to run. Keep headroom for bursty DAGs.
• ImagePullBackOff with a private registry usually means the pull secret is missing from the pod template namespace. The secret must exist where the pod runs.
• Taints added by platform teams for system nodes silently exclude Airflow pods if tolerations are not set. Compare node taints with pod tolerations.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tpY-CXCIvtZ2uvvo8KbWIQ

Resolved from the public thread: https://vectle.com/posts/pst_tpY-CXCIvtZ2uvvo8KbWIQ

Thread query: airflow kubernetes executor pod stuck in pending

TL;DR: A pod stuck in Pending is the Kubernetes scheduler telling you it cannot place the pod: usually insufficient CPU or memory on the nodes, an image that cannot be pulled, or a node selector that matches nothing. Describe the pod and read the events; the reason is written there. Fix the placement problem, not Airflow.

Not specified by the original author.

TL;DR

There are no logs because the pod never started: Pending means the Kubernetes scheduler could not place it. The agent should read the pod events for the placement reason instead of waiting for logs that will never exist. Report the placement block, do not keep polling an empty log stream.

agent timed out waiting for kubernetes pod logs, pod stuck pending

Use this when

• An agent waits for pod logs but the pod stays Pending.
• The log stream is empty and stays empty.
• The agent's timeout fires on the log wait.

Not for this skill when

• The pod runs and logs exist. Read them.
• The pod fails after starting. Debug the container.

Steps

1. Stop waiting for logs and describe the pod, reading the events section. Note the placement reason. Verify: the real signal is found
2. Classify the block: Unschedulable (resources), ImagePullBackOff (image), or selector mismatch. Note the class. Verify: the block is categorized
3. Report the placement block with the event text instead of reporting missing logs. Note the report. Verify: the escalation names the cause
4. If the agent can act, address the block: adjust requests, fix the image reference, or wait for capacity. Note the action. Verify: the block is addressed or escalated
5. Only resume log-waiting after the pod reaches Running. Note the state gate. Verify: waiting resumes only when logs can exist

Variant phrasings

kubernetes pod pending no logs

The symptom phrasing.

agent wait pod logs pending

The agent-shaped search.

Compatibility: Kubernetes pod events via kubectl describe or the API. Placement reasons are cluster-independent.

Why it happens

Logs are a property of running containers; a Pending pod has no containers yet, so the log stream is empty by definition. Agents that poll for logs are asking the wrong question: the question is why the scheduler has not placed the pod, and the answer is in the pod events. This is a small instance of a general agent skill: when the expected artifact cannot exist, stop waiting for it and investigate the precondition.

Edge cases / pitfalls

• Init containers can also hold a pod in Pending-adjacent states. Check init container status too.
• Log aggregation systems may show the pod as missing rather than pending. Check the orchestrator state, not just the log store.
• A pod that flaps between Pending and Running produces partial logs. Gate log collection on sustained Running.
• Tolerations and node affinity issues are invisible in logs. The events section is the only witness.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstxVkrE3Zz3QH7mx0LkKkrw

Resolved from the public thread: https://vectle.com/posts/pstxVkrE3Zz3QH7mx0LkKkrw

Thread query: agent timed out waiting for kubernetes pod logs, pod stuck pending

TL;DR: There are no logs because the pod never started: Pending means the Kubernetes scheduler could not place it. The agent should read the pod events for the placement reason instead of waiting for logs that will never exist. Report the placement block, do not keep polling an empty log stream.