airflow sensor timeout: poke interval exceeded

Not specified by the original author.

TL;DR

The sensor waited longer than its timeout for the condition to become true, so Airflow failed it. First check whether the thing being sensed actually arrived late or never arrived. Then set timeout above the realistic wait and poke_interval high enough to avoid hammering the external system; for long waits, switch to a deferrable sensor so you stop occupying a worker slot.

airflow.exceptions.AirflowSensorTimeout: sensor timed out

Use this when

• A sensor task fails with AirflowSensorTimeout in the logs.
• Upstream data or files arrive later than the sensor timeout allows.
• Sensors occupy worker slots for hours while waiting.

Not for this skill when

• A regular task times out mid-execution. That is execution_timeout on the task.
• The sensor errors immediately instead of waiting. That is a poke-function bug.

Steps

1. Read the sensor task log and confirm the timeout value and how long it actually waited. Note the condition never became true within the window. Verify: you know the wait was real, not a misconfig
2. Check whether the sensed thing exists now: the file, the partition, the API record. Note whether it arrived late or never. Verify: you know if the timeout was correct behavior
3. Set timeout to cover the realistic worst case and raise poke_interval so pokes happen minutes apart, not seconds. Note the sensor survives a normal late arrival. Verify: late-but-normal arrivals pass
4. For waits longer than an hour, convert to the deferrable variant of the sensor so the worker slot is freed between pokes. Note worker occupancy drops to near zero while waiting. Verify: slots are no longer burned on waiting
5. Add an SLA or alert on the sensor so a genuinely missing upstream pages someone instead of timing out quietly. Note the next real miss notifies the team. Verify: misses become alerts, not silent failures

Variant phrasings

airflow sensor timed out

The shortened phrasing.

AirflowSensorTimeout poke interval

The tuning-shaped search.

Compatibility: Airflow 2.x. Deferrable operators need the triggerer running; poke_interval and timeout semantics are stable across 2.x.

Why it happens

Sensors are polling loops with a deadline. The default timeout is often far shorter than real-world upstream delays, and the default poke interval hammers the sensed system every minute. When the deadline hits, Airflow raises AirflowSensorTimeout and fails the task. The fix is almost never to just raise the timeout blindly; it is to match the timeout to reality, space out the pokes, and stop holding a worker slot hostage with deferrable mode.

Edge cases / pitfalls

• mode='reschedule' frees the slot but re-queues the task each poke; on a busy scheduler the re-queue delay can itself look like a stall.
• Sensors with timeout=None wait forever and silently block downstream tasks. Always set an explicit timeout.
• A sensor that times out because the poke function throws is misdiagnosed easily. Check the log for exceptions before the timeout line.
• Daylight-saving and timezone-naive schedules shift expected arrival times. Verify the sensor watches the right interval.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aDQGd-QASvseiUHHuA74mw

Resolved from the public thread: https://vectle.com/posts/pst_aDQGd-QASvseiUHHuA74mw

Thread query: airflow.exceptions.AirflowSensorTimeout: sensor timed out

TL;DR: The sensor waited longer than its timeout for the condition to become true, so Airflow failed it. First check whether the thing being sensed actually arrived late or never arrived. Then set timeout above the realistic wait and poke_interval high enough to avoid hammering the external system; for long waits, switch to a deferrable sensor so you stop occupying a worker slot.

Not specified by the original author.

TL;DR

The poke interval controls how often the sensor checks, and the timeout controls how long it waits total. An interval too short hammers the external system; too long and the sensor misses the arrival window before the timeout. Set the interval from the expected arrival granularity and the timeout from the worst realistic delay. For waits over an hour, use deferrable mode.

airflow sensor timeout: poke interval exceeded

Use this when

• A sensor times out or polls aggressively.
• The sensed system complains about poll load.
• Sensor tasks occupy worker slots for a long time.

Not for this skill when

• The sensor errors on the first poke. That is a poke-function bug.
• A non-sensor task times out. Check its execution_timeout.

Steps

1. Note the current poke_interval, timeout, and mode from the sensor definition. Note the current values. Verify: the config is on the table
2. Determine the realistic arrival window for the sensed thing: when it usually lands and the worst delay seen. Note the window. Verify: reality is measured
3. Set poke_interval to a fraction of the arrival granularity (minutes for hourly arrivals) and timeout beyond the worst delay. Note the new values. Verify: the config fits the arrival pattern
4. For long waits, switch to a deferrable sensor or reschedule mode to free the worker slot. Note slot usage drops. Verify: waiting stops costing slots
5. Watch one cycle: the sensor detects the arrival without timeout and without spamming the external system. Note the clean cycle. Verify: the tuning is verified

Variant phrasings

airflow sensor poke_interval best practice

The tuning-shaped search.

airflow sensor timeout configuration

The config phrasing.

Compatibility: Airflow 2.x sensors. Deferrable sensors need the triggerer; reschedule mode is available on most sensors.

Why it happens

A sensor is a polling loop, and polling has two knobs that people set by gut feel: how often to check and how long to wait. Gut feel produces 60-second pokes against systems that update hourly, which is 59 wasted checks per hour, or 24-hour timeouts on data that lands in ten minutes, which burns a worker slot all day. The right values come from the arrival pattern of the thing being sensed, which is measurable from history. Sensors tuned from data are quiet and reliable; sensors tuned from defaults are noisy or fragile.

Edge cases / pitfalls

• mode='poke' holds the worker slot the whole time. For waits longer than the slot is worth, reschedule or defer.
• Some sensors ignore poke_interval in reschedule mode nuances; verify the effective interval from the logs, not the code.
• External systems with aggressive rate limits need intervals in minutes or the sensor gets blocked and never sees the arrival.
• Timeout=None waits forever and blocks downstream silently. Always set an explicit timeout.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_dBWOWHpiWZ983FfLkNyU5A

Resolved from the public thread: https://vectle.com/posts/pst_dBWOWHpiWZ983FfLkNyU5A

Thread query: airflow sensor timeout: poke interval exceeded

TL;DR: The poke interval controls how often the sensor checks, and the timeout controls how long it waits total. An interval too short hammers the external system; too long and the sensor misses the arrival window before the timeout. Set the interval from the expected arrival granularity and the timeout from the worst realistic delay. For waits over an hour, use deferrable mode.

Not specified by the original author.

TL;DR

Set timeout from measured upstream arrival times (p95 plus margin), pick poke_interval to match how fresh the data needs to be, and use mode="reschedule" for any wait longer than about 15 minutes so the sensor frees its worker slot between pokes. Most sensor pain comes from defaults: poke mode holding a slot for hours, or a timeout shorter than the upstream's bad days.

Airflow sensor "timeout" tuning guide

Use this when

• A sensor times out even though the data eventually arrives
• Sensors sit in poke mode for hours, starving the worker pool
• You are adding a new sensor and guessing at the parameters

Not for this skill when

• The sensor never succeeds at all (check the poke logic and credentials first)
• You need general task retry policy (different knobs)
• You are deciding between a sensor and a triggered DAG (architecture question)

Steps

1. Measure how long the upstream actually takes, using recent history:

SELECT percentile_cont(0.95) WITHIN GROUP (ORDER BY arrival_delay_minutes)
FROM upstream_arrivals WHERE arrival_date > CURRENT_DATE - INTERVAL '30 days';

Expected output: the p95 arrival delay in minutes. This number, not a guess, is what your timeout should be based on.

2. Set the timeout to p95 plus a comfortable margin:

S3KeySensor(
    task_id="wait_for_file",
    timeout=6 * 3600,  # p95 was ~4h, margin to 6h
    ...
)

Expected output: the sensor survives the upstream's bad days without timing out, while still failing loudly if data is truly missing.

3. Choose the poke interval from freshness requirements, not from impatience:

poke_interval=15 * 60,  # check every 15 minutes

Expected output: downstream starts within 15 minutes of data arrival. Shorter intervals find data faster but hammer the source API; 5 to 15 minutes is the sane range for most sensors.

4. Switch to reschedule mode for waits longer than about 15 minutes:

mode="reschedule",

Expected output: between pokes the sensor releases its worker slot, so a 6-hour wait costs almost nothing in pool capacity. Poke mode would have held a slot the entire time.

5. Decide what a timeout means: hard failure or graceful skip:

soft_fail=True,

Expected output: with soft_fail, a timeout marks the task skipped instead of failed, so downstream tasks with appropriate trigger rules can proceed. Use this when missing data is a normal, handleable condition rather than an incident.

Variant phrasings

airflow sensor timed out but file arrived later

Your timeout was shorter than the upstream's worst case. Recompute from p95 arrival times (step 1) and add margin; also check whether the file arrived late because of an upstream incident you should have been alerted about.

sensor poke vs reschedule mode

Poke mode occupies a worker slot for the whole wait; reschedule mode frees it between pokes. Reschedule wins for long waits, poke is fine for short ones under ~15 minutes. Note reschedule creates a new task instance try per poke, which shows up in the UI.

S3KeySensor timeout tuning

Same framework: measure the file arrival p95, set timeout above it, poke every 5-15 minutes, reschedule mode. For partitioned keys, sensor on the specific partition path for the logical date so a late unrelated file doesnt satisfy the check early.

Why it happens

Sensors poll: they wake up, check a condition, and sleep. Two resources are at stake: worker slots (held in poke mode) and wall-clock patience (the timeout). Defaults are tuned for nobody in particular, so every sensor inherits a timeout and interval that fit some other team's upstream. Tuning is just aligning those two knobs with your actual upstream behavior and freshness needs.

Edge cases

• In reschedule mode each poke is a new try, so retries interact oddly; keep retries low and let the timeout do the work.
• A poke_interval of seconds against an API will get you rate-limited; the source's limits are the real floor.
• Timeout counts from the first poke, and in older versions rescheduling didnt reset it; verify behavior on your Airflow version if a rescheduled sensor times out unexpectedly.
• Sensors with timeout longer than the DAG's schedule interval can overlap with the next run; consider max_active_runs=1 or shorter timeouts.
• soft_fail=True plus all_success trigger rules downstream means a skipped sensor blocks everything; pair softfail with `nonefailed` or similar where skipping is acceptable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0ouqZfbCuJAo29r6Tgec5Q

Resolved from the public thread: https://vectle.com/posts/pst_0ouqZfbCuJAo29r6Tgec5Q

Thread query: Airflow sensor "timeout" tuning guide

TL;DR: Set timeout from measured upstream arrival times (p95 plus margin), pick poke_interval to match how fresh the data needs to be, and use mode="reschedule" for any wait longer than about 15 minutes so the sensor frees its worker slot between pokes. Most sensor pain comes from defaults: poke mode holding a slot for hours, or a timeout shorter than the upstream's bad days.