airflow zombie tasks stuck in queued, scheduler not picking them up

Not specified by the original author.

TL;DR

Queued forever usually means the executor never got the task, the worker that had it died and its heartbeat stopped, or the pool and concurrency limits silently cap the DAG at zero effective slots. Check the task instance state history, the executor event log, and pool occupancy in that order. Clear a genuinely orphaned task instance only after confirming no worker process is running it, or you will get two copies of the same task writing at once.

airflow zombie tasks stuck in queued, scheduler not picking them up

Use this when

• Task instances sit in queued for far longer than the DAG normally waits.
• Workers are idle and report free slots while tasks stay queued.
• A worker restart or deploy left tasks behind in queued or running.

Not for this skill when

• The DAG is paused or the run is queued behind maxactiveruns. That is expected backpressure.
• The pool is full of genuinely running tasks. Wait or raise the pool deliberately.

Steps

1. Inspect one stuck task instance: its state history, try number, hostname, and external executor id. Note whether a worker ever claimed it. Verify: you know if the task was adopted or never sent
2. Check the scheduler and executor logs around the queue time for heartbeat loss, an executor event mismatch, or a worker pod that was killed. Note the event that orphaned it. Verify: the orphaning event is identified
3. Verify capacity: pool slots, the DAG maxactivetasks, worker concurrency, and queued tasks ahead of it. Note any limit sitting at zero free slots for this pool. Verify: a silent capacity cap is confirmed or ruled out
4. Confirm no process is running the task on the recorded hostname or pod, then clear that single task instance so the scheduler requeues it. Note the clear is scoped to the one instance. Verify: the task is requeued without a duplicate running copy
5. Watch the next scheduling loop and confirm the task moves to running and finishes. Note the state transition times. Verify: tasks flow through the executor again

Variant phrasings

airflow tasks stuck in queued zombie task

The zombie wording for the same orphan.

airflow scheduler not scheduling queued tasks

The scheduler-focused phrasing.

Compatibility: Airflow 2.x with Celery, Kubernetes, or Local executors. Zombie detection settings such as the scheduler zombie task threshold apply to running tasks whose heartbeats stopped; purely queued orphans are handled at the task instance level.

Why it happens

Airflow tracks work in two places that can disagree: task instance rows in the metadata database, and the executor's own queue of jobs it believes it owns. When a worker dies mid-flight, a deploy replaces pods, or the scheduler restarts between writing the row and handing off the job, a row can say queued while no executor anywhere holds the work. Nothing retries it because, from each component's view, someone else owns it. The scheduler zombie and orphan handling catches some of these, but rows that never reached a worker fall outside heartbeat based detection.

Edge cases / pitfalls

• Clearing a task that is actually running on a slow worker creates a duplicate execution. Always check the recorded host first.
• A pool set to zero slots, or a pool renamed in config while old rows keep the old name, queues tasks forever with no error anywhere.
• Kubernetes pods evicted during node drains leave exactly this pattern. Correlate queue times with cluster events before blaming Airflow.
• Repeated zombies after every deploy point at workers being killed before they finish. Lengthen the grace period instead of clearing forever.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZhEoHvA1FxbB83q6wKl-Hg

Resolved from the public thread: https://vectle.com/posts/pst_ZhEoHvA1FxbB83q6wKl-Hg

Thread query: airflow zombie tasks stuck in queued, scheduler not picking them up

TL;DR: Queued forever usually means the executor never got the task, the worker that had it died and its heartbeat stopped, or the pool and concurrency limits silently cap the DAG at zero effective slots. Check the task instance state history, the executor event log, and pool occupancy in that order. Clear a genuinely orphaned task instance only after confirming no worker process is running it, or you will get two copies of the same task writing at once.

Not specified by the original author.

TL;DR

Something outside Airflow killed the worker process, usually the kernel OOM killer or a node eviction, while the task instance still believed it was running. The fix is to find the kill reason in the worker or infrastructure logs, address it (usually more memory or eviction-proof scheduling), then clear the zombie task instance and rerun. The message is Airflow noticing the mismatch, not the cause.

Executor reports task instance finished (failed) although the task says its running. Was the task killed externally?

Use this when

• Task logs contain "was the task killed externally"
• A task shows as running forever but produces no new log lines
• This started after moving to KubernetesExecutor or after a cluster event

Not for this skill when

• Tasks are stuck in queued, scheduled, or upforretry (scheduler-side issue)
• The task failed normally with a traceback in its logs
• The scheduler itself is unhealthy or not heartbeating

Steps

1. Read the tail of the task log around the time it died:

airflow tasks logs my_dag my_task 2026-10-01 --log-level INFO | tail -50

Expected output: the last lines before death. An abrupt cutoff with no traceback points to an external kill; a Python traceback points to an in-task bug instead.

2. Check the infrastructure for the kill reason. On Kubernetes:

kubectl describe pod [worker pod] | grep -A5 "Last State"

Expected output: the last container state, often Reason: OOMKilled or Reason: Evicted. That line names your culprit.

3. If it was OOMKilled, either give the task more memory or make it use less:

MyOperator(
    task_id="my_task",
    executor_config={"KubernetesExecutor": {"request_memory": "2Gi", "limit_memory": "4Gi"}},
)

Expected output: the pod spec requests enough memory that the kernel stops killing it. Alternatively chunk the work so peak memory drops.

4. Clear the zombie task instance so the DAG can move on:

airflow tasks clear my_dag -t my_task -s 2026-10-01 -e 2026-10-01 --yes

Expected output: the stuck "running" state is cleared. The task becomes eligible to run again on the next scheduler loop.

5. Harden against recurrence: set realistic memory requests on memory-hungry tasks, add retries for preemption-prone node pools, and alert on the zombie message pattern:

default_args = {"retries": 2, "retry_delay": timedelta(minutes=10)}

Expected output: the occasional eviction becomes a quiet retry instead of a stuck DAG and a confused on-call.

Variant phrasings

was the task killed externally airflow

Yes, almost always. That message is the executor reporting that the process vanished without the task instance state changing. Start at the infrastructure layer, not the DAG code.

airflow task says running but finished

The DB row says running because nothing updated it when the process died. Clearing the task instance (step 4) resets the state; the scheduler will not do it on its own.

zombie task airflow kubernetes executor

The KubernetesExecutor is the most common source because pods get OOMKilled and evicted routinely. Memory requests and limits per task are the durable fix; the default pod spec fits nobody's workload.

Why it happens

Airflow tracks task state in two places: the metadata DB row ("running") and the actual worker process. The executor reconciles them by checking whether the process is alive. When the kernel or orchestrator kills the process from outside, the DB row is never updated, so the next reconciliation finds a "running" task with no living process and reports the mismatch. It is a symptom of infrastructure pressure, not an Airflow bug.

Edge cases

• With CeleryExecutor, a worker restart or SIGKILL produces the same symptom; check worker logs and the broker for disconnects.
• airflow jobs check can find scheduler and worker jobs that died without cleanup; run it if zombies recur.
• A task that forks subprocesses can leave orphans that hold resources after the kill; make sure cleanup handles child processes.
• Database locks or a full metadata disk can mimic zombie behavior by preventing state updates; check DB health if no kill reason appears anywhere.
• Marking the zombie as failed manually without clearing can confuse downstream trigger rules; prefer tasks clear so the task reruns cleanly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lEB9ikL2Wg3My8IAt9PHEw

Resolved from the public thread: https://vectle.com/posts/pst_lEB9ikL2Wg3My8IAt9PHEw

Thread query: "executor reports task instance finished" zombie tasks

TL;DR: Something outside Airflow killed the worker process, usually the kernel OOM killer or a node eviction, while the task instance still believed it was running. The fix is to find the kill reason in the worker or infrastructure logs, address it (usually more memory or eviction-proof scheduling), then clear the zombie task instance and rerun. The message is Airflow noticing the mismatch, not the cause.

Not specified by the original author.

TL;DR

Tasks are stuck in scheduled state because every slot in their pool is taken. Find which pool is full, see which tasks are holding the slots, then decide: raise the slot count, move tasks to another pool, or clear the stuck tasks that are squatting. The pool is a throttle doing its job, exhaustion means the throttle is engaged somewhere it should not be.

Airflow pool slots exhausted: tasks stuck in "scheduled" with no available pool slots

Use this when

• tasks sit in scheduled state and never start
• the UI shows a pool at zero open slots
• a new DAG's tasks never get picked up by workers

Not for this skill when

• tasks are queued but workers are idle, check the executor instead
• tasks fail on start, that is a task problem not a pool problem
• the scheduler itself is down, fix that first

Steps

1. List the pools and their slot usage to find the exhausted one:

airflow pools list

Expected output: each pool with total slots and running, queued, and open counts. The exhausted pool shows zero open slots while tasks wait.

2. See which tasks are holding slots in that pool right now:

SELECT dag_id, task_id, state, pool
FROM task_instance
WHERE pool = 'my_pool' AND state IN ('running', 'scheduled', 'queued')
ORDER BY start_date;

Expected output: the tasks occupying the pool. Long-running or stuck tasks near the top are your squatters, holding slots without doing useful work.

3. Check how long the running tasks have held their slots:

SELECT task_id, state,
       EXTRACT(EPOCH FROM (NOW() - start_date))/3600 AS hours_running
FROM task_instance
WHERE pool = 'my_pool' AND state = 'running'
ORDER BY start_date
LIMIT 10;

Expected output: hours each running task has held its slot. Anything running far past its normal duration is stuck and a candidate for clearing.

4. Raise the slot count if the pool is genuinely undersized for current demand:

airflow pools set my_pool 16 "increased for backfill"

Expected output: the pool now has 16 slots. Tasks waiting in scheduled state start getting picked up on the next scheduler loop.

5. Or clear the stuck tasks that are squatting on slots so the scheduler reschedules them fresh:

airflow tasks clear my_dag --task-regex my_stuck_task --only-running --yes

Expected output: the stuck task instances clear and release their slots. The scheduler picks them up again, and this time they either run or fail fast instead of hanging.

Variant phrasings

airflow tasks stuck in scheduled state

Scheduled means waiting on a slot, a dependency, or priority weight. Step 1 distinguishes pool exhaustion from the other causes quickly.

how to increase airflow pool slots

Step 4, the pools set command. Raise it deliberately, since the pool exists to protect something downstream from overload.

airflow default_pool exhausted

Everything lands in defaultpool unless assigned elsewhere. Either raise defaultpool or move heavy DAGs to dedicated pools so one greedy DAG cannot starve the rest.

Why it happens

Pools cap how many tasks can run concurrently to protect shared resources like a warehouse or an external API. When demand exceeds slots, extra tasks wait in scheduled state indefinitely. Exhaustion is usually growth (more tasks than the pool was sized for) or leakage (stuck tasks holding slots forever), and steps 2 and 3 tell you which one you are dealing with.

Edge cases

• Raising slots without checking downstream capacity just moves the overload to the warehouse or API. Size pools against the real bottleneck.
• default_pool is shared by everything. One greedy DAG can starve the rest, so give heavy DAGs their own pools.
• Deferred tasks can hold pool slots while waiting on triggers in some versions. Check for deferrable operators piling up.
• Pool changes apply on the next scheduler loop, not instantly. Give it a minute before re-diagnosing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstmg9q3V1HcvcvHo1F8Bddw

Resolved from the public thread: https://vectle.com/posts/pstmg9q3V1HcvcvHo1F8Bddw

Thread query: Airflow "pool slots" exhausted: how to debug

TL;DR: Tasks are stuck in scheduled state because every slot in their pool is taken. Find which pool is full, see which tasks are holding the slots, then decide: raise the slot count, move tasks to another pool, or clear the stuck tasks that are squatting. The pool is a throttle doing its job, exhaustion means the throttle is engaged somewhere it should not be.