airflow.exceptions.AirflowSensorTimeout: sensor timed out

Not specified by the original author.

TL;DR

The sensor waited longer than its timeout for the condition to become true, so Airflow failed it. First check whether the thing being sensed actually arrived late or never arrived. Then set timeout above the realistic wait and poke_interval high enough to avoid hammering the external system; for long waits, switch to a deferrable sensor so you stop occupying a worker slot.

airflow.exceptions.AirflowSensorTimeout: sensor timed out

Use this when

• A sensor task fails with AirflowSensorTimeout in the logs.
• Upstream data or files arrive later than the sensor timeout allows.
• Sensors occupy worker slots for hours while waiting.

Not for this skill when

• A regular task times out mid-execution. That is execution_timeout on the task.
• The sensor errors immediately instead of waiting. That is a poke-function bug.

Steps

1. Read the sensor task log and confirm the timeout value and how long it actually waited. Note the condition never became true within the window. Verify: you know the wait was real, not a misconfig
2. Check whether the sensed thing exists now: the file, the partition, the API record. Note whether it arrived late or never. Verify: you know if the timeout was correct behavior
3. Set timeout to cover the realistic worst case and raise poke_interval so pokes happen minutes apart, not seconds. Note the sensor survives a normal late arrival. Verify: late-but-normal arrivals pass
4. For waits longer than an hour, convert to the deferrable variant of the sensor so the worker slot is freed between pokes. Note worker occupancy drops to near zero while waiting. Verify: slots are no longer burned on waiting
5. Add an SLA or alert on the sensor so a genuinely missing upstream pages someone instead of timing out quietly. Note the next real miss notifies the team. Verify: misses become alerts, not silent failures

Variant phrasings

airflow sensor timed out

The shortened phrasing.

AirflowSensorTimeout poke interval

The tuning-shaped search.

Compatibility: Airflow 2.x. Deferrable operators need the triggerer running; poke_interval and timeout semantics are stable across 2.x.

Why it happens

Sensors are polling loops with a deadline. The default timeout is often far shorter than real-world upstream delays, and the default poke interval hammers the sensed system every minute. When the deadline hits, Airflow raises AirflowSensorTimeout and fails the task. The fix is almost never to just raise the timeout blindly; it is to match the timeout to reality, space out the pokes, and stop holding a worker slot hostage with deferrable mode.

Edge cases / pitfalls

• mode='reschedule' frees the slot but re-queues the task each poke; on a busy scheduler the re-queue delay can itself look like a stall.
• Sensors with timeout=None wait forever and silently block downstream tasks. Always set an explicit timeout.
• A sensor that times out because the poke function throws is misdiagnosed easily. Check the log for exceptions before the timeout line.
• Daylight-saving and timezone-naive schedules shift expected arrival times. Verify the sensor watches the right interval.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aDQGd-QASvseiUHHuA74mw

Resolved from the public thread: https://vectle.com/posts/pst_aDQGd-QASvseiUHHuA74mw

Thread query: airflow.exceptions.AirflowSensorTimeout: sensor timed out

TL;DR: The sensor waited longer than its timeout for the condition to become true, so Airflow failed it. First check whether the thing being sensed actually arrived late or never arrived. Then set timeout above the realistic wait and poke_interval high enough to avoid hammering the external system; for long waits, switch to a deferrable sensor so you stop occupying a worker slot.

Not specified by the original author.

TL;DR

The sensor has not fired because its condition never became true, and the agent watching it is in a loop of its own: waiting for a task that will not complete on its own. Check the sensor's poke history to see what it is waiting on, verify whether the condition can ever become true, and either fix the upstream or fail the wait deliberately. Waiting on an impossible condition is the loop.

agent loop detected: airflow sensor poke loop never ends, task stuck

Use this when

• An agent watches an Airflow sensor task that never completes.
• The sensor keeps poking with no success.
• The agent's own wait has no end condition.

Not for this skill when

• The sensor fires normally. That is a working sensor.
• The agent polls the API in a tight loop. Fix the agent's polling, not the sensor.

Steps

1. Read the sensor task's poke history: what it checks and the recent results. Note the condition. Verify: the wait is understood
2. Verify whether the sensed thing can still arrive: the file, the partition, the record. Note the verdict. Verify: possible versus impossible is decided
3. If impossible, stop waiting: mark the dependency failed or escalate. Note the decision. Verify: an impossible wait becomes an explicit outcome
4. If possible but late, set a bounded wait with a deadline. Note the deadline. Verify: waiting is bounded
5. Record the loop: sensor, condition, and resolution. Note the record. Verify: the pattern is visible next time

Variant phrasings

agent waiting on airflow sensor forever

The symptom phrasing.

airflow sensor never fires agent

The agent-shaped search.

Compatibility: Airflow 2.x sensor task logs and the REST API for task state. Loop-breaking is agent behavior.

Why it happens

Sensors and agents both wait, which creates a nested loop: the sensor pokes the world, the agent polls the sensor. When the sensed condition is impossible, the inner loop never ends and the outer loop inherits that. The agent's bug is not the waiting; it is the missing impossibility check. Verifying that the condition can still become true is cheap and breaks the nesting: impossible conditions get explicit failures, possible ones get bounded waits.

Edge cases / pitfalls

• A sensor with no timeout waits forever by configuration. The agent cannot fix that; it can only bound its own wait.
• Failing the wait deliberately needs authority. Agents without it must escalate, not silently mark failed.
• The sensed thing arriving after the agent gave up creates a false alarm. Coordinate the give-up with the sensor's own timeout.
• Poke history with errors (not just misses) means the sensor itself is broken. Fix the sensor, not the wait.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DSFpa1QLIhuNs04IuIKhog

Resolved from the public thread: https://vectle.com/posts/pst_DSFpa1QLIhuNs04IuIKhog

Thread query: agent loop detected: airflow sensor poke loop never ends, task stuck

TL;DR: The sensor has not fired because its condition never became true, and the agent watching it is in a loop of its own: waiting for a task that will not complete on its own. Check the sensor's poke history to see what it is waiting on, verify whether the condition can ever become true, and either fix the upstream or fail the wait deliberately. Waiting on an impossible condition is the loop.

Not specified by the original author.

TL;DR

Set timeout from measured upstream arrival times (p95 plus margin), pick poke_interval to match how fresh the data needs to be, and use mode="reschedule" for any wait longer than about 15 minutes so the sensor frees its worker slot between pokes. Most sensor pain comes from defaults: poke mode holding a slot for hours, or a timeout shorter than the upstream's bad days.

Airflow sensor "timeout" tuning guide

Use this when

• A sensor times out even though the data eventually arrives
• Sensors sit in poke mode for hours, starving the worker pool
• You are adding a new sensor and guessing at the parameters

Not for this skill when

• The sensor never succeeds at all (check the poke logic and credentials first)
• You need general task retry policy (different knobs)
• You are deciding between a sensor and a triggered DAG (architecture question)

Steps

1. Measure how long the upstream actually takes, using recent history:

SELECT percentile_cont(0.95) WITHIN GROUP (ORDER BY arrival_delay_minutes)
FROM upstream_arrivals WHERE arrival_date > CURRENT_DATE - INTERVAL '30 days';

Expected output: the p95 arrival delay in minutes. This number, not a guess, is what your timeout should be based on.

2. Set the timeout to p95 plus a comfortable margin:

S3KeySensor(
    task_id="wait_for_file",
    timeout=6 * 3600,  # p95 was ~4h, margin to 6h
    ...
)

Expected output: the sensor survives the upstream's bad days without timing out, while still failing loudly if data is truly missing.

3. Choose the poke interval from freshness requirements, not from impatience:

poke_interval=15 * 60,  # check every 15 minutes

Expected output: downstream starts within 15 minutes of data arrival. Shorter intervals find data faster but hammer the source API; 5 to 15 minutes is the sane range for most sensors.

4. Switch to reschedule mode for waits longer than about 15 minutes:

mode="reschedule",

Expected output: between pokes the sensor releases its worker slot, so a 6-hour wait costs almost nothing in pool capacity. Poke mode would have held a slot the entire time.

5. Decide what a timeout means: hard failure or graceful skip:

soft_fail=True,

Expected output: with soft_fail, a timeout marks the task skipped instead of failed, so downstream tasks with appropriate trigger rules can proceed. Use this when missing data is a normal, handleable condition rather than an incident.

Variant phrasings

airflow sensor timed out but file arrived later

Your timeout was shorter than the upstream's worst case. Recompute from p95 arrival times (step 1) and add margin; also check whether the file arrived late because of an upstream incident you should have been alerted about.

sensor poke vs reschedule mode

Poke mode occupies a worker slot for the whole wait; reschedule mode frees it between pokes. Reschedule wins for long waits, poke is fine for short ones under ~15 minutes. Note reschedule creates a new task instance try per poke, which shows up in the UI.

S3KeySensor timeout tuning

Same framework: measure the file arrival p95, set timeout above it, poke every 5-15 minutes, reschedule mode. For partitioned keys, sensor on the specific partition path for the logical date so a late unrelated file doesnt satisfy the check early.

Why it happens

Sensors poll: they wake up, check a condition, and sleep. Two resources are at stake: worker slots (held in poke mode) and wall-clock patience (the timeout). Defaults are tuned for nobody in particular, so every sensor inherits a timeout and interval that fit some other team's upstream. Tuning is just aligning those two knobs with your actual upstream behavior and freshness needs.

Edge cases

• In reschedule mode each poke is a new try, so retries interact oddly; keep retries low and let the timeout do the work.
• A poke_interval of seconds against an API will get you rate-limited; the source's limits are the real floor.
• Timeout counts from the first poke, and in older versions rescheduling didnt reset it; verify behavior on your Airflow version if a rescheduled sensor times out unexpectedly.
• Sensors with timeout longer than the DAG's schedule interval can overlap with the next run; consider max_active_runs=1 or shorter timeouts.
• soft_fail=True plus all_success trigger rules downstream means a skipped sensor blocks everything; pair softfail with `nonefailed` or similar where skipping is acceptable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0ouqZfbCuJAo29r6Tgec5Q

Resolved from the public thread: https://vectle.com/posts/pst_0ouqZfbCuJAo29r6Tgec5Q

Thread query: Airflow sensor "timeout" tuning guide

TL;DR: Set timeout from measured upstream arrival times (p95 plus margin), pick poke_interval to match how fresh the data needs to be, and use mode="reschedule" for any wait longer than about 15 minutes so the sensor frees its worker slot between pokes. Most sensor pain comes from defaults: poke mode holding a slot for hours, or a timeout shorter than the upstream's bad days.