airflow zombie tasks stuck in queued, scheduler not picking them up

TL;DR

Queued forever usually means the executor never got the task, the worker that had it died and its heartbeat stopped, or the pool and concurrency limits silently cap the DAG at zero effective slots. Check the task instance state history, the executor event log, and pool occupancy in that order. Clear a genuinely orphaned task instance only after confirming no worker process is running it, or you will get two copies of the same task writing at once.

airflow zombie tasks stuck in queued, scheduler not picking them up

Use this when

• Task instances sit in queued for far longer than the DAG normally waits.
• Workers are idle and report free slots while tasks stay queued.
• A worker restart or deploy left tasks behind in queued or running.

Not for this skill when

• The DAG is paused or the run is queued behind maxactiveruns. That is expected backpressure.
• The pool is full of genuinely running tasks. Wait or raise the pool deliberately.

Steps

1. Inspect one stuck task instance: its state history, try number, hostname, and external executor id. Note whether a worker ever claimed it. Verify: you know if the task was adopted or never sent
2. Check the scheduler and executor logs around the queue time for heartbeat loss, an executor event mismatch, or a worker pod that was killed. Note the event that orphaned it. Verify: the orphaning event is identified
3. Verify capacity: pool slots, the DAG maxactivetasks, worker concurrency, and queued tasks ahead of it. Note any limit sitting at zero free slots for this pool. Verify: a silent capacity cap is confirmed or ruled out
4. Confirm no process is running the task on the recorded hostname or pod, then clear that single task instance so the scheduler requeues it. Note the clear is scoped to the one instance. Verify: the task is requeued without a duplicate running copy
5. Watch the next scheduling loop and confirm the task moves to running and finishes. Note the state transition times. Verify: tasks flow through the executor again

Variant phrasings

airflow tasks stuck in queued zombie task

The zombie wording for the same orphan.

airflow scheduler not scheduling queued tasks

The scheduler-focused phrasing.

Compatibility: Airflow 2.x with Celery, Kubernetes, or Local executors. Zombie detection settings such as the scheduler zombie task threshold apply to running tasks whose heartbeats stopped; purely queued orphans are handled at the task instance level.

Why it happens

Airflow tracks work in two places that can disagree: task instance rows in the metadata database, and the executor's own queue of jobs it believes it owns. When a worker dies mid-flight, a deploy replaces pods, or the scheduler restarts between writing the row and handing off the job, a row can say queued while no executor anywhere holds the work. Nothing retries it because, from each component's view, someone else owns it. The scheduler zombie and orphan handling catches some of these, but rows that never reached a worker fall outside heartbeat based detection.

Edge cases / pitfalls

• Clearing a task that is actually running on a slow worker creates a duplicate execution. Always check the recorded host first.
• A pool set to zero slots, or a pool renamed in config while old rows keep the old name, queues tasks forever with no error anywhere.
• Kubernetes pods evicted during node drains leave exactly this pattern. Correlate queue times with cluster events before blaming Airflow.
• Repeated zombies after every deploy point at workers being killed before they finish. Lengthen the grace period instead of clearing forever.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZhEoHvA1FxbB83q6wKl-Hg