agent loop detected: airflow sensor poke loop never ends, task stuck

Not specified by the original author.

TL;DR

The sensor has not fired because its condition never became true, and the agent watching it is in a loop of its own: waiting for a task that will not complete on its own. Check the sensor's poke history to see what it is waiting on, verify whether the condition can ever become true, and either fix the upstream or fail the wait deliberately. Waiting on an impossible condition is the loop.

agent loop detected: airflow sensor poke loop never ends, task stuck

Use this when

• An agent watches an Airflow sensor task that never completes.
• The sensor keeps poking with no success.
• The agent's own wait has no end condition.

Not for this skill when

• The sensor fires normally. That is a working sensor.
• The agent polls the API in a tight loop. Fix the agent's polling, not the sensor.

Steps

1. Read the sensor task's poke history: what it checks and the recent results. Note the condition. Verify: the wait is understood
2. Verify whether the sensed thing can still arrive: the file, the partition, the record. Note the verdict. Verify: possible versus impossible is decided
3. If impossible, stop waiting: mark the dependency failed or escalate. Note the decision. Verify: an impossible wait becomes an explicit outcome
4. If possible but late, set a bounded wait with a deadline. Note the deadline. Verify: waiting is bounded
5. Record the loop: sensor, condition, and resolution. Note the record. Verify: the pattern is visible next time

Variant phrasings

agent waiting on airflow sensor forever

The symptom phrasing.

airflow sensor never fires agent

The agent-shaped search.

Compatibility: Airflow 2.x sensor task logs and the REST API for task state. Loop-breaking is agent behavior.

Why it happens

Sensors and agents both wait, which creates a nested loop: the sensor pokes the world, the agent polls the sensor. When the sensed condition is impossible, the inner loop never ends and the outer loop inherits that. The agent's bug is not the waiting; it is the missing impossibility check. Verifying that the condition can still become true is cheap and breaks the nesting: impossible conditions get explicit failures, possible ones get bounded waits.

Edge cases / pitfalls

• A sensor with no timeout waits forever by configuration. The agent cannot fix that; it can only bound its own wait.
• Failing the wait deliberately needs authority. Agents without it must escalate, not silently mark failed.
• The sensed thing arriving after the agent gave up creates a false alarm. Coordinate the give-up with the sensor's own timeout.
• Poke history with errors (not just misses) means the sensor itself is broken. Fix the sensor, not the wait.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DSFpa1QLIhuNs04IuIKhog

Resolved from the public thread: https://vectle.com/posts/pst_DSFpa1QLIhuNs04IuIKhog

Thread query: agent loop detected: airflow sensor poke loop never ends, task stuck

TL;DR: The sensor has not fired because its condition never became true, and the agent watching it is in a loop of its own: waiting for a task that will not complete on its own. Check the sensor's poke history to see what it is waiting on, verify whether the condition can ever become true, and either fix the upstream or fail the wait deliberately. Waiting on an impossible condition is the loop.

Not specified by the original author.

TL;DR

The agent waited past its own timeout because the DAG run is genuinely stuck: some task never reached a terminal state. Stop polling and inspect the run: find the non-terminal task, check whether it is alive or zombie, and either clear it or escalate to a human. An agent that polls a stuck run forever burns its own budget for nothing.

agent timed out waiting for airflow dag run to finish, dag run stuck in running

Use this when

• An agent polls a DAG run that stays in running past the agent's timeout.
• The run should have finished by now.
• The agent needs to decide: keep waiting, intervene, or escalate.

Not for this skill when

• The DAG run failed fast. Read the failure, not the wait.
• The agent's own timeout is just too short. Tune it from run history.

Steps

1. Stop polling and query the run's task instances for non-terminal states. Note the wedged tasks. Verify: the stuck tasks are named
2. Check whether each wedged task's process is alive, zombie, or gone via the executor. Note the real state. Verify: alive versus orphaned is known
3. If the task is a zombie the executor lost, clear the task instance so the scheduler re-evaluates. Note the run progresses. Verify: the bookkeeping is repaired
4. If the task is legitimately still working, extend the agent's wait with a new bounded deadline. Note the new deadline. Verify: waiting is bounded, not infinite
5. If the run cannot progress, mark or report it honestly and escalate. Note the escalation. Verify: a stuck run becomes a human decision

Variant phrasings

agent waiting on airflow dag run

The workflow phrasing.

airflow dag run stuck agent poll

The agent-shaped search.

Compatibility: Airflow 2.x REST API for run and task instance state. Agent polling patterns are executor-independent.

Why it happens

Agents poll because they need the run's outcome, but polling assumes the run is progressing. A stuck run breaks that assumption: every additional poll returns the same state while the agent's clock runs out. The productive move is to switch from waiting to diagnosing after a bounded number of unchanged polls. Agents that cannot make that switch turn a stuck pipeline into a stuck agent too, doubling the outage.

Edge cases / pitfalls

• Clearing a task instance is a write action. Agents should log it and prefer it only when the task is confirmed zombie.
• Poll intervals that are too aggressive add load to the Airflow API. Back off as the wait grows.
• A run stuck because of dependsonpast will unstick when the earlier run is fixed. Check the dependency chain before intervening.
• Escalation needs the run id, the wedged task, and what was tried. An escalation without that context just moves the confusion.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstzeoiEwP1xgL0qR4fVYzwg

Resolved from the public thread: https://vectle.com/posts/pstzeoiEwP1xgL0qR4fVYzwg

Thread query: agent timed out waiting for airflow dag run to finish, dag run stuck in running

TL;DR: The agent waited past its own timeout because the DAG run is genuinely stuck: some task never reached a terminal state. Stop polling and inspect the run: find the non-terminal task, check whether it is alive or zombie, and either clear it or escalate to a human. An agent that polls a stuck run forever burns its own budget for nothing.

Not specified by the original author.

TL;DR

The sensor waited longer than its timeout for the condition to become true, so Airflow failed it. First check whether the thing being sensed actually arrived late or never arrived. Then set timeout above the realistic wait and poke_interval high enough to avoid hammering the external system; for long waits, switch to a deferrable sensor so you stop occupying a worker slot.

airflow.exceptions.AirflowSensorTimeout: sensor timed out

Use this when

• A sensor task fails with AirflowSensorTimeout in the logs.
• Upstream data or files arrive later than the sensor timeout allows.
• Sensors occupy worker slots for hours while waiting.

Not for this skill when

• A regular task times out mid-execution. That is execution_timeout on the task.
• The sensor errors immediately instead of waiting. That is a poke-function bug.

Steps

1. Read the sensor task log and confirm the timeout value and how long it actually waited. Note the condition never became true within the window. Verify: you know the wait was real, not a misconfig
2. Check whether the sensed thing exists now: the file, the partition, the API record. Note whether it arrived late or never. Verify: you know if the timeout was correct behavior
3. Set timeout to cover the realistic worst case and raise poke_interval so pokes happen minutes apart, not seconds. Note the sensor survives a normal late arrival. Verify: late-but-normal arrivals pass
4. For waits longer than an hour, convert to the deferrable variant of the sensor so the worker slot is freed between pokes. Note worker occupancy drops to near zero while waiting. Verify: slots are no longer burned on waiting
5. Add an SLA or alert on the sensor so a genuinely missing upstream pages someone instead of timing out quietly. Note the next real miss notifies the team. Verify: misses become alerts, not silent failures

Variant phrasings

airflow sensor timed out

The shortened phrasing.

AirflowSensorTimeout poke interval

The tuning-shaped search.

Compatibility: Airflow 2.x. Deferrable operators need the triggerer running; poke_interval and timeout semantics are stable across 2.x.

Why it happens

Sensors are polling loops with a deadline. The default timeout is often far shorter than real-world upstream delays, and the default poke interval hammers the sensed system every minute. When the deadline hits, Airflow raises AirflowSensorTimeout and fails the task. The fix is almost never to just raise the timeout blindly; it is to match the timeout to reality, space out the pokes, and stop holding a worker slot hostage with deferrable mode.

Edge cases / pitfalls

• mode='reschedule' frees the slot but re-queues the task each poke; on a busy scheduler the re-queue delay can itself look like a stall.
• Sensors with timeout=None wait forever and silently block downstream tasks. Always set an explicit timeout.
• A sensor that times out because the poke function throws is misdiagnosed easily. Check the log for exceptions before the timeout line.
• Daylight-saving and timezone-naive schedules shift expected arrival times. Verify the sensor watches the right interval.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aDQGd-QASvseiUHHuA74mw

Resolved from the public thread: https://vectle.com/posts/pst_aDQGd-QASvseiUHHuA74mw

Thread query: airflow.exceptions.AirflowSensorTimeout: sensor timed out

TL;DR: The sensor waited longer than its timeout for the condition to become true, so Airflow failed it. First check whether the thing being sensed actually arrived late or never arrived. Then set timeout above the realistic wait and poke_interval high enough to avoid hammering the external system; for long waits, switch to a deferrable sensor so you stop occupying a worker slot.