agent timed out waiting for airflow dag run to finish, dag run stuck in running

TL;DR

The agent waited past its own timeout because the DAG run is genuinely stuck: some task never reached a terminal state. Stop polling and inspect the run: find the non-terminal task, check whether it is alive or zombie, and either clear it or escalate to a human. An agent that polls a stuck run forever burns its own budget for nothing.

agent timed out waiting for airflow dag run to finish, dag run stuck in running

Use this when

• An agent polls a DAG run that stays in running past the agent's timeout.
• The run should have finished by now.
• The agent needs to decide: keep waiting, intervene, or escalate.

Not for this skill when

• The DAG run failed fast. Read the failure, not the wait.
• The agent's own timeout is just too short. Tune it from run history.

Steps

1. Stop polling and query the run's task instances for non-terminal states. Note the wedged tasks. Verify: the stuck tasks are named
2. Check whether each wedged task's process is alive, zombie, or gone via the executor. Note the real state. Verify: alive versus orphaned is known
3. If the task is a zombie the executor lost, clear the task instance so the scheduler re-evaluates. Note the run progresses. Verify: the bookkeeping is repaired
4. If the task is legitimately still working, extend the agent's wait with a new bounded deadline. Note the new deadline. Verify: waiting is bounded, not infinite
5. If the run cannot progress, mark or report it honestly and escalate. Note the escalation. Verify: a stuck run becomes a human decision

Variant phrasings

agent waiting on airflow dag run

The workflow phrasing.

airflow dag run stuck agent poll

The agent-shaped search.

Compatibility: Airflow 2.x REST API for run and task instance state. Agent polling patterns are executor-independent.

Why it happens

Agents poll because they need the run's outcome, but polling assumes the run is progressing. A stuck run breaks that assumption: every additional poll returns the same state while the agent's clock runs out. The productive move is to switch from waiting to diagnosing after a bounded number of unchanged polls. Agents that cannot make that switch turn a stuck pipeline into a stuck agent too, doubling the outage.

Edge cases / pitfalls

• Clearing a task instance is a write action. Agents should log it and prefer it only when the task is confirmed zombie.
• Poll intervals that are too aggressive add load to the Airflow API. Back off as the wait grows.
• A run stuck because of dependsonpast will unstick when the earlier run is fixed. Check the dependency chain before intervening.
• Escalation needs the run id, the wedged task, and what was tried. An escalation without that context just moves the confusion.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstzeoiEwP1xgL0qR4fVYzwg