airflow dag run stuck in running state

TL;DR

A DAG run stays running when at least one task never reaches a terminal state: a zombie task the executor lost track of, a sensor waiting forever, or a task stuck in queued with no worker to pick it up. Find the non-terminal tasks in the UI, fix or clear the wedged one, and the run completes. If the task is truly dead, mark it failed so downstream knows.

airflow dag run stuck in running state

Use this when

• A DAG run shows running long after it should have finished.
• All visible tasks look done but the run never closes.
• The next scheduled run is blocked behind the stuck one.

Not for this skill when

• A single task is slow but progressing. That is a performance problem.
• The run is waiting on a sensor with a long timeout. That is by design until the timeout.

Steps

1. Open the grid view and list every task instance not in success, failed, or skipped. Note the exact wedged tasks. Verify: the stuck tasks are named
2. Check the wedged task's logs and the executor: is the process alive, zombie, or gone? Note the real state. Verify: you know if the task is alive or orphaned
3. For zombie tasks the executor lost, clear or mark the task instance so the scheduler can re-evaluate it. Note the task reaches a terminal state. Verify: the run can progress again
4. For sensors waiting forever, check whether the sensed condition will ever become true; if not, fail the sensor deliberately. Note downstream handles the failure. Verify: an impossible wait becomes an explicit failure
5. If the run is unrecoverable, mark the run failed or succeeded manually and document why. Note the next scheduled run proceeds. Verify: scheduling resumes

Variant phrasings

airflow dagrun stuck running

The compact phrasing.

airflow dag run never finishes

The symptom search.

Compatibility: Airflow 2.x, all executors. Zombie detection behavior differs between LocalExecutor and Celery/Kubernetes executors.

Why it happens

A DAG run is running until every task instance is terminal, and Airflow trusts the executor to report task state. When that reporting breaks, a killed worker, a lost Celery result, a pod evicted without notice, the task instance stays in a non-terminal state forever and the run with it. The run is not doing anything; it is waiting on bookkeeping that will never arrive. Clearing the wedged task instance lets the scheduler recompute the run state honestly.

Edge cases / pitfalls

• Clearing a task that actually wrote partial results can cause duplicates on retry. Check what the task did before clearing.
• dependsonpast with a stuck run blocks every future run of that DAG. Fix the stuck run before the backlog grows.
• Marking a run success when tasks actually failed lies to downstream consumers. Prefer failed with a note.
• Kubernetes executor pods stuck in Terminating hold task instances in running. Delete the pod to release the task.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstmFUeR-NTokYgr7-j49AUQ