airflow deferrable operators triggerer config

Not specified by the original author.

TL;DR

Run the triggerer (airflow triggerer), then use deferrable operators (e.g. TimeDeltaSensorAsync, deferrable provider operators) so waiting tasks release their worker slot and resume on trigger. Deferrable turns "occupy a worker while waiting" into "sleep until the event."

airflow deferrable operators triggerer config

Use this when

• Sensors sit for hours occupying worker slots
• You are setting up the triggerer for the first time
• You want operators that wait without holding workers

Not for this skill when

• You need dynamic task mapping
• You need SLA miss alerts
• DAGs do not schedule

Steps

1. Start the triggerer. It is a separate Airflow component:

airflow triggerer

Expected output: a running triggerer process. Without it, deferrable tasks defer and never wake up; this is the most common "my async sensor hangs forever" cause.

2. Use the deferrable (async) variants of waiting operators:

from airflow.providers.standard.sensors.time_delta import TimeDeltaSensorAsync
wait = TimeDeltaSensorAsync(task_id="wait_2h", delta=timedelta(hours=2))

Expected output: the task defers instead of occupying a worker for 2 hours. Most providers ship async variants; the naming pattern is *Async or a deferrable=True flag.

3. Convert existing operators where the provider supports it:

# many provider operators accept deferrable=True:
MyWaitOperator(task_id="wait", deferrable=True)

Expected output: the same operator logic, but waiting happens in the triggerer. Check the provider docs for your operator; not every operator has a deferrable path.

4. Size the triggerer for your deferred load:

[triggerer]
# capacity = how many deferred tasks one triggerer handles
capacity = 1000

Expected output: headroom for your deferred task count. One triggerer with default capacity handles roughly a thousand concurrent deferred tasks; beyond that, run more triggerer replicas.

5. Monitor deferred tasks separately from running ones:

In the UI, deferred tasks show as light-blue "deferred" state,
distinct from "running". If deferred tasks pile up and never
resume, the triggerer is down or at capacity (steps 1, 4).

Expected output: the ability to tell "waiting efficiently" from "stuck." Alert on triggerer process health like any other Airflow component.

Variant phrasings

airflow triggerer not running

Start it (step 1). Deferrable tasks without a triggerer wait forever; the scheduler does not run triggers itself.

airflow sensor occupying worker slot

Switch to the async/deferrable sensor variant (steps 2-3). Classic sensors poll in the worker; deferrable ones sleep in the triggerer.

airflow deferrable vs sensor poke mode

Poke mode occupies a worker and polls; reschedule mode frees the worker between pokes; deferrable frees the worker entirely until the trigger fires. Deferrable is the most efficient when supported.

Why it happens

A classic sensor occupies a worker slot for its whole wait, so 100 hourly sensors need 100 worker slots doing nothing. Deferrable operators hand the waiting to the triggerer, an async event loop that can watch thousands of triggers in one process, and the worker slot is freed for real work.

Edge cases

• The triggerer needs network access to whatever the trigger watches; a triggerer in a restricted subnet fails where workers succeeded.
• Triggerer HA: run multiple triggerers; deferred tasks are claimed by whichever triggerer picks them up.
• Not all providers support deferrable mode; check before refactoring a DAG around it.
• Deferred tasks still consume a DB row and scheduler attention; they are cheap, not free.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lDdhPk9-6f3WdKDe3c8eMQ

Resolved from the public thread: https://vectle.com/posts/pst_lDdhPk9-6f3WdKDe3c8eMQ

Thread query: airflow deferrable operators triggerer config

TL;DR: Run the triggerer (airflow triggerer), then use deferrable operators (e.g. TimeDeltaSensorAsync, deferrable provider operators) so waiting tasks release their worker slot and resume on trigger. Deferrable turns "occupy a worker while waiting" into "sleep until the event."

Not specified by the original author.

TL;DR

A deferred task is not waiting on the worker at all. It handed a trigger object to the triggerer service, which polls cheaply until the condition is met, then wakes the task. Stuck in deferred almost always means the triggerer process is not running, cannot import the trigger class, or has hit its capacity. Check triggerer health and its logs for the trigger id before touching the sensor itself.

airflow deferrable sensor stuck in deferred state, triggerer not picking up task

Use this when

• A deferrable sensor or operator sits in deferred far past the expected wait.
• Many unrelated deferred tasks across DAGs are stuck at the same time.
• The task log shows it deferred successfully and then nothing happens.

Not for this skill when

• A poke mode sensor holds a worker slot and times out. That sensor never deferred.
• The external condition genuinely has not happened yet. Check the source system first.

Steps

1. Confirm the triggerer service is running: check the process or deployment and its heartbeat in the Airflow UI or the trigger table. Note whether any triggerer has checked in recently. Verify: triggerer liveness is known
2. Read the triggerer logs for the trigger id tied to your task instance and note import errors or repeated exceptions for that trigger class. Note the exact error. Verify: a broken trigger class is confirmed or ruled out
3. Check trigger capacity: how many triggers are registered versus the triggerer capacity setting, and whether one noisy DAG registered thousands. Note the counts. Verify: capacity exhaustion is confirmed or ruled out
4. If the trigger class fails to import on the triggerer, install the same provider package there that the workers use, and restart the triggerer. Note the trigger resumes logging. Verify: the triggerer runs the same code as the workers
5. If the trigger row is corrupt or wedged, clear the task instance so it re-defers with a fresh trigger, then watch it leave deferred when the condition lands. Note the wake-up in the task log. Verify: the task resumes and completes

Variant phrasings

airflow deferred task never resumes triggerer down

The triggerer-down phrasing.

airflow deferrable operator stuck deferred

The operator version of the same stall.

Compatibility: Airflow 2.2 and later with deferrable operators and a running triggerer; Airflow 3 keeps the same model. Trigger capacity defaults differ by version, so check your own configuration value.

Why it happens

Deferrable operators exist to stop sensors burning worker slots while waiting. The task serializes a small trigger object, exits, and the triggerer, a separate asyncio service, polls many triggers in one process. That design moves the failure surface: the worker and scheduler can be perfectly healthy while every deferred task in the system is frozen because the one component nobody monitors, the triggerer, is down, misconfigured, or missing a provider package its triggers need.

Edge cases / pitfalls

• One triggerer is a single point of failure for waits. Run at least two in production; they share work through the trigger table.
• A trigger that raises immediately on every poll spams the triggerer log and can starve other triggers. Fix the trigger code, not the sensor timeout.
• Clearing a deferred task cancels its trigger. If the external job the trigger watched keeps running, expect to reconcile it separately.
• Custom triggers must be importable from a path the triggerer sees. Code that lives only in the DAG folder packaging of workers may not be there.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_J7pxVQtoulhrafa92kAwhw

Resolved from the public thread: https://vectle.com/posts/pst_J7pxVQtoulhrafa92kAwhw

Thread query: airflow deferrable sensor stuck in deferred state, triggerer not picking up task

TL;DR: A deferred task is not waiting on the worker at all. It handed a trigger object to the triggerer service, which polls cheaply until the condition is met, then wakes the task. Stuck in deferred almost always means the triggerer process is not running, cannot import the trigger class, or has hit its capacity. Check triggerer health and its logs for the trigger id before touching the sensor itself.

Not specified by the original author.

TL;DR

If deferred tasks hang in 3.2.1 while the triggerer appears healthy, do not trust the heartbeat: upgrade to 3.2.2 or later, where the lock-based serialization was replaced with response multiplexing so concurrent trigger requests no longer contend or deadlock.

Steps

1. If deferred tasks hang in 3.2.1 while the triggerer appears healthy, do not trust the heartbeat: upgrade to 3.2.2 or later, where the lock-based serialization was replaced with response multiplexing so concurrent trigger requests no longer contend or deadlock. As a diagnostic, check whether the number of processed triggers is actually advancing and whether deferred tasks start moving right after a triggerer restart; a triggerer that heartbeats but makes no progress matches this bug. Also keep custom logging config classes out of the triggerer (AIRFLOWLOGGINGLOGGINGCONFIGCLASS empty) since the reporter's custom logging handler interacted with the failure.

When to use

You are seeing this: Also keep custom logging config classes out of the triggerer (AIRFLOWLOGGINGLOGGINGCONFIGCLASS empty) since the reporter's custom logging handler interacted with the failure. Use this skill when you run into "Airflow 3.2.1 triggerer heartbeats but silently processes zero triggers".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 3.2.1, 3.2.2. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Also keep custom logging config classes out of the triggerer (AIRFLOWLOGGINGLOGGINGCONFIGCLASS empty) since the reporter's custom logging handler interacted with the failure.

Source: GitHub issue apache/airflow#65945 (closed, 11 comments): after upgrading to Airflow 3.2.1 the triggerer sent heartbeats and looked healthy to the scheduler, but silently processed zero triggers, so every deferred task stalled until it timed out. Root cause was a triggerer race condition and deadlock: triggers calling synchronous SDK methods could crash the triggerer's internal subprocess, and the partial fix in 3.2.1 introduced a new deadlock with the same visible symptom under load.

Verified resolution: If deferred tasks hang in 3.2.1 while the triggerer appears healthy, do not trust the heartbeat: upgrade to 3.2.2 or later, where the lock-based serialization was replaced with response multiplexing so concurrent trigger requests no longer contend or deadlock. As a diagnostic, check whether the number of processed triggers is actually advancing and whether deferred tasks start moving right after a triggerer restart; a triggerer that heartbeats but makes no progress matches this bug. Also keep custom logging config classes out of the triggerer (AIRFLOWLOGGINGLOGGINGCONFIGCLASS empty) since the reporter's custom logging handler interacted with the failure.

Source URL: https://github.com/apache/airflow/issues/65945