airflow kubernetes executor pod stuck in pending

TL;DR

A pod stuck in Pending is the Kubernetes scheduler telling you it cannot place the pod: usually insufficient CPU or memory on the nodes, an image that cannot be pulled, or a node selector that matches nothing. Describe the pod and read the events; the reason is written there. Fix the placement problem, not Airflow.

airflow kubernetes executor pod stuck in pending

Use this when

• Airflow task pods stay in Pending and tasks never start.
• It started after a resource request change or a cluster scale-down.
• Some tasks schedule fine while others hang in Pending.

Not for this skill when

• Pods start and then fail. That is a task or image problem.
• Tasks are stuck in queued in Airflow. Check the Airflow scheduler first.

Steps

1. Describe the pending pod and read the events section. Note the exact reason: Unschedulable, ImagePullBackOff, or similar. Verify: the scheduler's complaint is in hand
2. For Unschedulable, compare the pod's CPU and memory requests against available node capacity. Note the shortfall. Verify: the resource gap is quantified
3. For image problems, verify the image name, tag, and pull secrets from a node that can reach the registry. Note the pull succeeds manually. Verify: image access is confirmed or the error is reproduced
4. Check node selectors, affinities, and taints on the pod spec against actual node labels. Note any selector that matches zero nodes. Verify: placement constraints are validated
5. Fix the shortfall: lower requests, add node capacity, or correct the image and selectors. Note new pods reach Running. Verify: placement succeeds

Variant phrasings

airflow k8s executor pod pending

The abbreviated search.

airflow task pod unschedulable

The event-shaped phrasing.

Compatibility: Airflow 2.x with KubernetesExecutor on any Kubernetes 1.24+. Pod spec comes from the executor config and pod templates.

Why it happens

The KubernetesExecutor delegates placement entirely to the Kubernetes scheduler, and Pending is that scheduler's waiting room. Airflow did its job by creating the pod; everything after that is cluster capacity and configuration. Data teams hit this when task resource requests grow organically, nobody watches node headroom, and one day the cluster is full. The pod events always name the reason, which makes this one of the more honest failures in the stack.

Edge cases / pitfalls

• Requests set far above actual usage waste capacity and cause Pending during peaks. Right-size with real usage data.
• Cluster autoscalers take minutes to add nodes. Short tasks can Pending longer than they would take to run. Keep headroom for bursty DAGs.
• ImagePullBackOff with a private registry usually means the pull secret is missing from the pod template namespace. The secret must exist where the pod runs.
• Taints added by platform teams for system nodes silently exclude Airflow pods if tolerations are not set. Compare node taints with pod tolerations.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tpY-CXCIvtZ2uvvo8KbWIQ

Resolved from the public thread: https://vectle.com/posts/pst_tpY-CXCIvtZ2uvvo8KbWIQ

Thread query: airflow kubernetes executor pod stuck in pending

TL;DR: A pod stuck in Pending is the Kubernetes scheduler telling you it cannot place the pod: usually insufficient CPU or memory on the nodes, an image that cannot be pulled, or a node selector that matches nothing. Describe the pod and read the events; the reason is written there. Fix the placement problem, not Airflow.