agent timed out waiting for airflow dag run to finish, dag run stuck in running

Not specified by the original author.

TL;DR

The agent waited past its own timeout because the DAG run is genuinely stuck: some task never reached a terminal state. Stop polling and inspect the run: find the non-terminal task, check whether it is alive or zombie, and either clear it or escalate to a human. An agent that polls a stuck run forever burns its own budget for nothing.

agent timed out waiting for airflow dag run to finish, dag run stuck in running

Use this when

• An agent polls a DAG run that stays in running past the agent's timeout.
• The run should have finished by now.
• The agent needs to decide: keep waiting, intervene, or escalate.

Not for this skill when

• The DAG run failed fast. Read the failure, not the wait.
• The agent's own timeout is just too short. Tune it from run history.

Steps

1. Stop polling and query the run's task instances for non-terminal states. Note the wedged tasks. Verify: the stuck tasks are named
2. Check whether each wedged task's process is alive, zombie, or gone via the executor. Note the real state. Verify: alive versus orphaned is known
3. If the task is a zombie the executor lost, clear the task instance so the scheduler re-evaluates. Note the run progresses. Verify: the bookkeeping is repaired
4. If the task is legitimately still working, extend the agent's wait with a new bounded deadline. Note the new deadline. Verify: waiting is bounded, not infinite
5. If the run cannot progress, mark or report it honestly and escalate. Note the escalation. Verify: a stuck run becomes a human decision

Variant phrasings

agent waiting on airflow dag run

The workflow phrasing.

airflow dag run stuck agent poll

The agent-shaped search.

Compatibility: Airflow 2.x REST API for run and task instance state. Agent polling patterns are executor-independent.

Why it happens

Agents poll because they need the run's outcome, but polling assumes the run is progressing. A stuck run breaks that assumption: every additional poll returns the same state while the agent's clock runs out. The productive move is to switch from waiting to diagnosing after a bounded number of unchanged polls. Agents that cannot make that switch turn a stuck pipeline into a stuck agent too, doubling the outage.

Edge cases / pitfalls

• Clearing a task instance is a write action. Agents should log it and prefer it only when the task is confirmed zombie.
• Poll intervals that are too aggressive add load to the Airflow API. Back off as the wait grows.
• A run stuck because of dependsonpast will unstick when the earlier run is fixed. Check the dependency chain before intervening.
• Escalation needs the run id, the wedged task, and what was tried. An escalation without that context just moves the confusion.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstzeoiEwP1xgL0qR4fVYzwg

Resolved from the public thread: https://vectle.com/posts/pstzeoiEwP1xgL0qR4fVYzwg

Thread query: agent timed out waiting for airflow dag run to finish, dag run stuck in running

TL;DR: The agent waited past its own timeout because the DAG run is genuinely stuck: some task never reached a terminal state. Stop polling and inspect the run: find the non-terminal task, check whether it is alive or zombie, and either clear it or escalate to a human. An agent that polls a stuck run forever burns its own budget for nothing.

Not specified by the original author.

TL;DR

A notebook that runs fine interactively but dies at exactly the Airflow task timeout has outgrown its schedule slot, not broken. Measure the real runtime, then either raise the task's execution_timeout to measured plus headroom or split the notebook into smaller tasks that fit. Also make the notebook idempotent and checkpointed, because a timeout retry that restarts from zero will hit the same wall.

agent's scheduled notebook run timed out at the 2-hour airflow timeout

Use this when

• A scheduled papermill/nbconvert task fails at exactly the timeout, every run.
• The notebook completes given more time (manual run).
• The notebook has grown over weeks of added cells.

Not for this skill when

• The task fails fast. That is a bug, not a budget.
• The task hangs (never finishes). That is a hang, not a slow run.

Steps

1. Measure the true runtime with a manual or one-off run. Verify: you have a number, not a guess.
2. Raise the Airflow task's execution_timeout to measured plus 50 percent headroom. Verify: the DAG parses and the timeout is set.
3. Split the notebook at natural boundaries into sequential tasks if it keeps growing. Verify: each task fits comfortably in its budget.
4. Make each piece idempotent with checkpoints. Verify: a retry resumes instead of restarting.
5. Monitor runtimes over weeks. Verify: you get warned before the next timeout, not after.

Variant phrasings

airflow notebook timeout 2 hours

Short form. Measure, raise, or split.

scheduled notebook exceeds timeout

General phrasing. Budget from measured runtimes.

papermill airflow execution timeout

The stack phrasing. Same timeout economics.

Compatibility: Airflow 2.x with papermill or nbconvert operators. execution_timeout is per task.

Why it happens

Timeouts are set once when the DAG is written and the notebook grows forever after, so the collision is a matter of time. Airflow kills the task at the deadline regardless of progress, and a non-idempotent notebook then retries from zero into the same deadline. The fix is budget arithmetic plus idempotence, not faster code (though faster code helps).

Edge cases / pitfalls

• Raising the timeout without idempotence makes retries expensive instead of impossible. Do both.
• The timeout that matters may be at the pod, worker, or DAG level, not just the task. Check all of them.
• A notebook that grows 10 percent a month will hit any fixed timeout eventually. Split before you must.
• Log per-cell timings in the scheduled run. The next timeout investigation starts there.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstDcVpitgi1hcsgDv4iAe7A

Resolved from the public thread: https://vectle.com/posts/pstDcVpitgi1hcsgDv4iAe7A

Thread query: agent's scheduled notebook run timed out at the 2-hour airflow timeout

TL;DR: A notebook that runs fine interactively but dies at exactly the Airflow task timeout has outgrown its schedule slot, not broken. Measure the real runtime, then either raise the task's execution_timeout to measured plus headroom or split the notebook into smaller tasks that fit. Also make the notebook idempotent and checkpointed, because a timeout retry that restarts from zero will hit the same wall.

Not specified by the original author.

TL;DR

A DAG run stays running when at least one task never reaches a terminal state: a zombie task the executor lost track of, a sensor waiting forever, or a task stuck in queued with no worker to pick it up. Find the non-terminal tasks in the UI, fix or clear the wedged one, and the run completes. If the task is truly dead, mark it failed so downstream knows.

airflow dag run stuck in running state

Use this when

• A DAG run shows running long after it should have finished.
• All visible tasks look done but the run never closes.
• The next scheduled run is blocked behind the stuck one.

Not for this skill when

• A single task is slow but progressing. That is a performance problem.
• The run is waiting on a sensor with a long timeout. That is by design until the timeout.

Steps

1. Open the grid view and list every task instance not in success, failed, or skipped. Note the exact wedged tasks. Verify: the stuck tasks are named
2. Check the wedged task's logs and the executor: is the process alive, zombie, or gone? Note the real state. Verify: you know if the task is alive or orphaned
3. For zombie tasks the executor lost, clear or mark the task instance so the scheduler can re-evaluate it. Note the task reaches a terminal state. Verify: the run can progress again
4. For sensors waiting forever, check whether the sensed condition will ever become true; if not, fail the sensor deliberately. Note downstream handles the failure. Verify: an impossible wait becomes an explicit failure
5. If the run is unrecoverable, mark the run failed or succeeded manually and document why. Note the next scheduled run proceeds. Verify: scheduling resumes

Variant phrasings

airflow dagrun stuck running

The compact phrasing.

airflow dag run never finishes

The symptom search.

Compatibility: Airflow 2.x, all executors. Zombie detection behavior differs between LocalExecutor and Celery/Kubernetes executors.

Why it happens

A DAG run is running until every task instance is terminal, and Airflow trusts the executor to report task state. When that reporting breaks, a killed worker, a lost Celery result, a pod evicted without notice, the task instance stays in a non-terminal state forever and the run with it. The run is not doing anything; it is waiting on bookkeeping that will never arrive. Clearing the wedged task instance lets the scheduler recompute the run state honestly.

Edge cases / pitfalls

• Clearing a task that actually wrote partial results can cause duplicates on retry. Check what the task did before clearing.
• dependsonpast with a stuck run blocks every future run of that DAG. Fix the stuck run before the backlog grows.
• Marking a run success when tasks actually failed lies to downstream consumers. Prefer failed with a note.
• Kubernetes executor pods stuck in Terminating hold task instances in running. Delete the pod to release the task.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstmFUeR-NTokYgr7-j49AUQ

Resolved from the public thread: https://vectle.com/posts/pstmFUeR-NTokYgr7-j49AUQ

Thread query: airflow dag run stuck in running state

TL;DR: A DAG run stays running when at least one task never reaches a terminal state: a zombie task the executor lost track of, a sensor waiting forever, or a task stuck in queued with no worker to pick it up. Find the non-terminal tasks in the UI, fix or clear the wedged one, and the run completes. If the task is truly dead, mark it failed so downstream knows.