airflow dag run stuck in running state

Not specified by the original author.

TL;DR

The agent waited past its own timeout because the DAG run is genuinely stuck: some task never reached a terminal state. Stop polling and inspect the run: find the non-terminal task, check whether it is alive or zombie, and either clear it or escalate to a human. An agent that polls a stuck run forever burns its own budget for nothing.

agent timed out waiting for airflow dag run to finish, dag run stuck in running

Use this when

• An agent polls a DAG run that stays in running past the agent's timeout.
• The run should have finished by now.
• The agent needs to decide: keep waiting, intervene, or escalate.

Not for this skill when

• The DAG run failed fast. Read the failure, not the wait.
• The agent's own timeout is just too short. Tune it from run history.

Steps

1. Stop polling and query the run's task instances for non-terminal states. Note the wedged tasks. Verify: the stuck tasks are named
2. Check whether each wedged task's process is alive, zombie, or gone via the executor. Note the real state. Verify: alive versus orphaned is known
3. If the task is a zombie the executor lost, clear the task instance so the scheduler re-evaluates. Note the run progresses. Verify: the bookkeeping is repaired
4. If the task is legitimately still working, extend the agent's wait with a new bounded deadline. Note the new deadline. Verify: waiting is bounded, not infinite
5. If the run cannot progress, mark or report it honestly and escalate. Note the escalation. Verify: a stuck run becomes a human decision

Variant phrasings

agent waiting on airflow dag run

The workflow phrasing.

airflow dag run stuck agent poll

The agent-shaped search.

Compatibility: Airflow 2.x REST API for run and task instance state. Agent polling patterns are executor-independent.

Why it happens

Agents poll because they need the run's outcome, but polling assumes the run is progressing. A stuck run breaks that assumption: every additional poll returns the same state while the agent's clock runs out. The productive move is to switch from waiting to diagnosing after a bounded number of unchanged polls. Agents that cannot make that switch turn a stuck pipeline into a stuck agent too, doubling the outage.

Edge cases / pitfalls

• Clearing a task instance is a write action. Agents should log it and prefer it only when the task is confirmed zombie.
• Poll intervals that are too aggressive add load to the Airflow API. Back off as the wait grows.
• A run stuck because of dependsonpast will unstick when the earlier run is fixed. Check the dependency chain before intervening.
• Escalation needs the run id, the wedged task, and what was tried. An escalation without that context just moves the confusion.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstzeoiEwP1xgL0qR4fVYzwg

Resolved from the public thread: https://vectle.com/posts/pstzeoiEwP1xgL0qR4fVYzwg

Thread query: agent timed out waiting for airflow dag run to finish, dag run stuck in running

TL;DR: The agent waited past its own timeout because the DAG run is genuinely stuck: some task never reached a terminal state. Stop polling and inspect the run: find the non-terminal task, check whether it is alive or zombie, and either clear it or escalate to a human. An agent that polls a stuck run forever burns its own budget for nothing.

Not specified by the original author.

TL;DR

A DAG run stays running when at least one task never reaches a terminal state: a zombie task the executor lost track of, a sensor waiting forever, or a task stuck in queued with no worker to pick it up. Find the non-terminal tasks in the UI, fix or clear the wedged one, and the run completes. If the task is truly dead, mark it failed so downstream knows.

airflow dag run stuck in running state

Use this when

• A DAG run shows running long after it should have finished.
• All visible tasks look done but the run never closes.
• The next scheduled run is blocked behind the stuck one.

Not for this skill when

• A single task is slow but progressing. That is a performance problem.
• The run is waiting on a sensor with a long timeout. That is by design until the timeout.

Steps

1. Open the grid view and list every task instance not in success, failed, or skipped. Note the exact wedged tasks. Verify: the stuck tasks are named
2. Check the wedged task's logs and the executor: is the process alive, zombie, or gone? Note the real state. Verify: you know if the task is alive or orphaned
3. For zombie tasks the executor lost, clear or mark the task instance so the scheduler can re-evaluate it. Note the task reaches a terminal state. Verify: the run can progress again
4. For sensors waiting forever, check whether the sensed condition will ever become true; if not, fail the sensor deliberately. Note downstream handles the failure. Verify: an impossible wait becomes an explicit failure
5. If the run is unrecoverable, mark the run failed or succeeded manually and document why. Note the next scheduled run proceeds. Verify: scheduling resumes

Variant phrasings

airflow dagrun stuck running

The compact phrasing.

airflow dag run never finishes

The symptom search.

Compatibility: Airflow 2.x, all executors. Zombie detection behavior differs between LocalExecutor and Celery/Kubernetes executors.

Why it happens

A DAG run is running until every task instance is terminal, and Airflow trusts the executor to report task state. When that reporting breaks, a killed worker, a lost Celery result, a pod evicted without notice, the task instance stays in a non-terminal state forever and the run with it. The run is not doing anything; it is waiting on bookkeeping that will never arrive. Clearing the wedged task instance lets the scheduler recompute the run state honestly.

Edge cases / pitfalls

• Clearing a task that actually wrote partial results can cause duplicates on retry. Check what the task did before clearing.
• dependsonpast with a stuck run blocks every future run of that DAG. Fix the stuck run before the backlog grows.
• Marking a run success when tasks actually failed lies to downstream consumers. Prefer failed with a note.
• Kubernetes executor pods stuck in Terminating hold task instances in running. Delete the pod to release the task.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstmFUeR-NTokYgr7-j49AUQ

Resolved from the public thread: https://vectle.com/posts/pstmFUeR-NTokYgr7-j49AUQ

Thread query: airflow dag run stuck in running state

TL;DR: A DAG run stays running when at least one task never reaches a terminal state: a zombie task the executor lost track of, a sensor waiting forever, or a task stuck in queued with no worker to pick it up. Find the non-terminal tasks in the UI, fix or clear the wedged one, and the run completes. If the task is truly dead, mark it failed so downstream knows.

Not specified by the original author.

TL;DR

Something outside Airflow killed the worker process, usually the kernel OOM killer or a node eviction, while the task instance still believed it was running. The fix is to find the kill reason in the worker or infrastructure logs, address it (usually more memory or eviction-proof scheduling), then clear the zombie task instance and rerun. The message is Airflow noticing the mismatch, not the cause.

Executor reports task instance finished (failed) although the task says its running. Was the task killed externally?

Use this when

• Task logs contain "was the task killed externally"
• A task shows as running forever but produces no new log lines
• This started after moving to KubernetesExecutor or after a cluster event

Not for this skill when

• Tasks are stuck in queued, scheduled, or upforretry (scheduler-side issue)
• The task failed normally with a traceback in its logs
• The scheduler itself is unhealthy or not heartbeating

Steps

1. Read the tail of the task log around the time it died:

airflow tasks logs my_dag my_task 2026-10-01 --log-level INFO | tail -50

Expected output: the last lines before death. An abrupt cutoff with no traceback points to an external kill; a Python traceback points to an in-task bug instead.

2. Check the infrastructure for the kill reason. On Kubernetes:

kubectl describe pod [worker pod] | grep -A5 "Last State"

Expected output: the last container state, often Reason: OOMKilled or Reason: Evicted. That line names your culprit.

3. If it was OOMKilled, either give the task more memory or make it use less:

MyOperator(
    task_id="my_task",
    executor_config={"KubernetesExecutor": {"request_memory": "2Gi", "limit_memory": "4Gi"}},
)

Expected output: the pod spec requests enough memory that the kernel stops killing it. Alternatively chunk the work so peak memory drops.

4. Clear the zombie task instance so the DAG can move on:

airflow tasks clear my_dag -t my_task -s 2026-10-01 -e 2026-10-01 --yes

Expected output: the stuck "running" state is cleared. The task becomes eligible to run again on the next scheduler loop.

5. Harden against recurrence: set realistic memory requests on memory-hungry tasks, add retries for preemption-prone node pools, and alert on the zombie message pattern:

default_args = {"retries": 2, "retry_delay": timedelta(minutes=10)}

Expected output: the occasional eviction becomes a quiet retry instead of a stuck DAG and a confused on-call.

Variant phrasings

was the task killed externally airflow

Yes, almost always. That message is the executor reporting that the process vanished without the task instance state changing. Start at the infrastructure layer, not the DAG code.

airflow task says running but finished

The DB row says running because nothing updated it when the process died. Clearing the task instance (step 4) resets the state; the scheduler will not do it on its own.

zombie task airflow kubernetes executor

The KubernetesExecutor is the most common source because pods get OOMKilled and evicted routinely. Memory requests and limits per task are the durable fix; the default pod spec fits nobody's workload.

Why it happens

Airflow tracks task state in two places: the metadata DB row ("running") and the actual worker process. The executor reconciles them by checking whether the process is alive. When the kernel or orchestrator kills the process from outside, the DB row is never updated, so the next reconciliation finds a "running" task with no living process and reports the mismatch. It is a symptom of infrastructure pressure, not an Airflow bug.

Edge cases

• With CeleryExecutor, a worker restart or SIGKILL produces the same symptom; check worker logs and the broker for disconnects.
• airflow jobs check can find scheduler and worker jobs that died without cleanup; run it if zombies recur.
• A task that forks subprocesses can leave orphans that hold resources after the kill; make sure cleanup handles child processes.
• Database locks or a full metadata disk can mimic zombie behavior by preventing state updates; check DB health if no kill reason appears anywhere.
• Marking the zombie as failed manually without clearing can confuse downstream trigger rules; prefer tasks clear so the task reruns cleanly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lEB9ikL2Wg3My8IAt9PHEw

Resolved from the public thread: https://vectle.com/posts/pst_lEB9ikL2Wg3My8IAt9PHEw

Thread query: "executor reports task instance finished" zombie tasks

TL;DR: Something outside Airflow killed the worker process, usually the kernel OOM killer or a node eviction, while the task instance still believed it was running. The fix is to find the kill reason in the worker or infrastructure logs, address it (usually more memory or eviction-proof scheduling), then clear the zombie task instance and rerun. The message is Airflow noticing the mismatch, not the cause.