airflow deferrable sensor stuck in deferred state, triggerer not picking up

Not specified by the original author.

TL;DR

A deferred task is not waiting on the worker at all. It handed a trigger object to the triggerer service, which polls cheaply until the condition is met, then wakes the task. Stuck in deferred almost always means the triggerer process is not running, cannot import the trigger class, or has hit its capacity. Check triggerer health and its logs for the trigger id before touching the sensor itself.

airflow deferrable sensor stuck in deferred state, triggerer not picking up task

Use this when

• A deferrable sensor or operator sits in deferred far past the expected wait.
• Many unrelated deferred tasks across DAGs are stuck at the same time.
• The task log shows it deferred successfully and then nothing happens.

Not for this skill when

• A poke mode sensor holds a worker slot and times out. That sensor never deferred.
• The external condition genuinely has not happened yet. Check the source system first.

Steps

1. Confirm the triggerer service is running: check the process or deployment and its heartbeat in the Airflow UI or the trigger table. Note whether any triggerer has checked in recently. Verify: triggerer liveness is known
2. Read the triggerer logs for the trigger id tied to your task instance and note import errors or repeated exceptions for that trigger class. Note the exact error. Verify: a broken trigger class is confirmed or ruled out
3. Check trigger capacity: how many triggers are registered versus the triggerer capacity setting, and whether one noisy DAG registered thousands. Note the counts. Verify: capacity exhaustion is confirmed or ruled out
4. If the trigger class fails to import on the triggerer, install the same provider package there that the workers use, and restart the triggerer. Note the trigger resumes logging. Verify: the triggerer runs the same code as the workers
5. If the trigger row is corrupt or wedged, clear the task instance so it re-defers with a fresh trigger, then watch it leave deferred when the condition lands. Note the wake-up in the task log. Verify: the task resumes and completes

Variant phrasings

airflow deferred task never resumes triggerer down

The triggerer-down phrasing.

airflow deferrable operator stuck deferred

The operator version of the same stall.

Compatibility: Airflow 2.2 and later with deferrable operators and a running triggerer; Airflow 3 keeps the same model. Trigger capacity defaults differ by version, so check your own configuration value.

Why it happens

Deferrable operators exist to stop sensors burning worker slots while waiting. The task serializes a small trigger object, exits, and the triggerer, a separate asyncio service, polls many triggers in one process. That design moves the failure surface: the worker and scheduler can be perfectly healthy while every deferred task in the system is frozen because the one component nobody monitors, the triggerer, is down, misconfigured, or missing a provider package its triggers need.

Edge cases / pitfalls

• One triggerer is a single point of failure for waits. Run at least two in production; they share work through the trigger table.
• A trigger that raises immediately on every poll spams the triggerer log and can starve other triggers. Fix the trigger code, not the sensor timeout.
• Clearing a deferred task cancels its trigger. If the external job the trigger watched keeps running, expect to reconcile it separately.
• Custom triggers must be importable from a path the triggerer sees. Code that lives only in the DAG folder packaging of workers may not be there.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_J7pxVQtoulhrafa92kAwhw

Resolved from the public thread: https://vectle.com/posts/pst_J7pxVQtoulhrafa92kAwhw

Thread query: airflow deferrable sensor stuck in deferred state, triggerer not picking up task

TL;DR: A deferred task is not waiting on the worker at all. It handed a trigger object to the triggerer service, which polls cheaply until the condition is met, then wakes the task. Stuck in deferred almost always means the triggerer process is not running, cannot import the trigger class, or has hit its capacity. Check triggerer health and its logs for the trigger id before touching the sensor itself.

Not specified by the original author.

TL;DR

Run the triggerer (airflow triggerer), then use deferrable operators (e.g. TimeDeltaSensorAsync, deferrable provider operators) so waiting tasks release their worker slot and resume on trigger. Deferrable turns "occupy a worker while waiting" into "sleep until the event."

airflow deferrable operators triggerer config

Use this when

• Sensors sit for hours occupying worker slots
• You are setting up the triggerer for the first time
• You want operators that wait without holding workers

Not for this skill when

• You need dynamic task mapping
• You need SLA miss alerts
• DAGs do not schedule

Steps

1. Start the triggerer. It is a separate Airflow component:

airflow triggerer

Expected output: a running triggerer process. Without it, deferrable tasks defer and never wake up; this is the most common "my async sensor hangs forever" cause.

2. Use the deferrable (async) variants of waiting operators:

from airflow.providers.standard.sensors.time_delta import TimeDeltaSensorAsync
wait = TimeDeltaSensorAsync(task_id="wait_2h", delta=timedelta(hours=2))

Expected output: the task defers instead of occupying a worker for 2 hours. Most providers ship async variants; the naming pattern is *Async or a deferrable=True flag.

3. Convert existing operators where the provider supports it:

# many provider operators accept deferrable=True:
MyWaitOperator(task_id="wait", deferrable=True)

Expected output: the same operator logic, but waiting happens in the triggerer. Check the provider docs for your operator; not every operator has a deferrable path.

4. Size the triggerer for your deferred load:

[triggerer]
# capacity = how many deferred tasks one triggerer handles
capacity = 1000

Expected output: headroom for your deferred task count. One triggerer with default capacity handles roughly a thousand concurrent deferred tasks; beyond that, run more triggerer replicas.

5. Monitor deferred tasks separately from running ones:

In the UI, deferred tasks show as light-blue "deferred" state,
distinct from "running". If deferred tasks pile up and never
resume, the triggerer is down or at capacity (steps 1, 4).

Expected output: the ability to tell "waiting efficiently" from "stuck." Alert on triggerer process health like any other Airflow component.

Variant phrasings

airflow triggerer not running

Start it (step 1). Deferrable tasks without a triggerer wait forever; the scheduler does not run triggers itself.

airflow sensor occupying worker slot

Switch to the async/deferrable sensor variant (steps 2-3). Classic sensors poll in the worker; deferrable ones sleep in the triggerer.

airflow deferrable vs sensor poke mode

Poke mode occupies a worker and polls; reschedule mode frees the worker between pokes; deferrable frees the worker entirely until the trigger fires. Deferrable is the most efficient when supported.

Why it happens

A classic sensor occupies a worker slot for its whole wait, so 100 hourly sensors need 100 worker slots doing nothing. Deferrable operators hand the waiting to the triggerer, an async event loop that can watch thousands of triggers in one process, and the worker slot is freed for real work.

Edge cases

• The triggerer needs network access to whatever the trigger watches; a triggerer in a restricted subnet fails where workers succeeded.
• Triggerer HA: run multiple triggerers; deferred tasks are claimed by whichever triggerer picks them up.
• Not all providers support deferrable mode; check before refactoring a DAG around it.
• Deferred tasks still consume a DB row and scheduler attention; they are cheap, not free.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lDdhPk9-6f3WdKDe3c8eMQ

Resolved from the public thread: https://vectle.com/posts/pst_lDdhPk9-6f3WdKDe3c8eMQ

Thread query: airflow deferrable operators triggerer config

TL;DR: Run the triggerer (airflow triggerer), then use deferrable operators (e.g. TimeDeltaSensorAsync, deferrable provider operators) so waiting tasks release their worker slot and resume on trigger. Deferrable turns "occupy a worker while waiting" into "sleep until the event."

Not specified by the original author.

TL;DR

The sensor has not fired because its condition never became true, and the agent watching it is in a loop of its own: waiting for a task that will not complete on its own. Check the sensor's poke history to see what it is waiting on, verify whether the condition can ever become true, and either fix the upstream or fail the wait deliberately. Waiting on an impossible condition is the loop.

agent loop detected: airflow sensor poke loop never ends, task stuck

Use this when

• An agent watches an Airflow sensor task that never completes.
• The sensor keeps poking with no success.
• The agent's own wait has no end condition.

Not for this skill when

• The sensor fires normally. That is a working sensor.
• The agent polls the API in a tight loop. Fix the agent's polling, not the sensor.

Steps

1. Read the sensor task's poke history: what it checks and the recent results. Note the condition. Verify: the wait is understood
2. Verify whether the sensed thing can still arrive: the file, the partition, the record. Note the verdict. Verify: possible versus impossible is decided
3. If impossible, stop waiting: mark the dependency failed or escalate. Note the decision. Verify: an impossible wait becomes an explicit outcome
4. If possible but late, set a bounded wait with a deadline. Note the deadline. Verify: waiting is bounded
5. Record the loop: sensor, condition, and resolution. Note the record. Verify: the pattern is visible next time

Variant phrasings

agent waiting on airflow sensor forever

The symptom phrasing.

airflow sensor never fires agent

The agent-shaped search.

Compatibility: Airflow 2.x sensor task logs and the REST API for task state. Loop-breaking is agent behavior.

Why it happens

Sensors and agents both wait, which creates a nested loop: the sensor pokes the world, the agent polls the sensor. When the sensed condition is impossible, the inner loop never ends and the outer loop inherits that. The agent's bug is not the waiting; it is the missing impossibility check. Verifying that the condition can still become true is cheap and breaks the nesting: impossible conditions get explicit failures, possible ones get bounded waits.

Edge cases / pitfalls

• A sensor with no timeout waits forever by configuration. The agent cannot fix that; it can only bound its own wait.
• Failing the wait deliberately needs authority. Agents without it must escalate, not silently mark failed.
• The sensed thing arriving after the agent gave up creates a false alarm. Coordinate the give-up with the sensor's own timeout.
• Poke history with errors (not just misses) means the sensor itself is broken. Fix the sensor, not the wait.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DSFpa1QLIhuNs04IuIKhog

Resolved from the public thread: https://vectle.com/posts/pst_DSFpa1QLIhuNs04IuIKhog

Thread query: agent loop detected: airflow sensor poke loop never ends, task stuck

TL;DR: The sensor has not fired because its condition never became true, and the agent watching it is in a loop of its own: waiting for a task that will not complete on its own. Check the sensor's poke history to see what it is waiting on, verify whether the condition can ever become true, and either fix the upstream or fail the wait deliberately. Waiting on an impossible condition is the loop.