airflow deferrable operators triggerer config

TL;DR

Run the triggerer (airflow triggerer), then use deferrable operators (e.g. TimeDeltaSensorAsync, deferrable provider operators) so waiting tasks release their worker slot and resume on trigger. Deferrable turns "occupy a worker while waiting" into "sleep until the event."

airflow deferrable operators triggerer config

Use this when

• Sensors sit for hours occupying worker slots
• You are setting up the triggerer for the first time
• You want operators that wait without holding workers

Not for this skill when

• You need dynamic task mapping
• You need SLA miss alerts
• DAGs do not schedule

Steps

1. Start the triggerer. It is a separate Airflow component:

airflow triggerer

Expected output: a running triggerer process. Without it, deferrable tasks defer and never wake up; this is the most common "my async sensor hangs forever" cause.

1. Use the deferrable (async) variants of waiting operators:

from airflow.providers.standard.sensors.time_delta import TimeDeltaSensorAsync
wait = TimeDeltaSensorAsync(task_id="wait_2h", delta=timedelta(hours=2))

Expected output: the task defers instead of occupying a worker for 2 hours. Most providers ship async variants; the naming pattern is *Async or a deferrable=True flag.

1. Convert existing operators where the provider supports it:

# many provider operators accept deferrable=True:
MyWaitOperator(task_id="wait", deferrable=True)

Expected output: the same operator logic, but waiting happens in the triggerer. Check the provider docs for your operator; not every operator has a deferrable path.

1. Size the triggerer for your deferred load:

[triggerer]
# capacity = how many deferred tasks one triggerer handles
capacity = 1000

Expected output: headroom for your deferred task count. One triggerer with default capacity handles roughly a thousand concurrent deferred tasks; beyond that, run more triggerer replicas.

1. Monitor deferred tasks separately from running ones:

In the UI, deferred tasks show as light-blue "deferred" state,
distinct from "running". If deferred tasks pile up and never
resume, the triggerer is down or at capacity (steps 1, 4).

Expected output: the ability to tell "waiting efficiently" from "stuck." Alert on triggerer process health like any other Airflow component.

Variant phrasings

airflow triggerer not running

Start it (step 1). Deferrable tasks without a triggerer wait forever; the scheduler does not run triggers itself.

airflow sensor occupying worker slot

Switch to the async/deferrable sensor variant (steps 2-3). Classic sensors poll in the worker; deferrable ones sleep in the triggerer.

airflow deferrable vs sensor poke mode

Poke mode occupies a worker and polls; reschedule mode frees the worker between pokes; deferrable frees the worker entirely until the trigger fires. Deferrable is the most efficient when supported.

Why it happens

A classic sensor occupies a worker slot for its whole wait, so 100 hourly sensors need 100 worker slots doing nothing. Deferrable operators hand the waiting to the triggerer, an async event loop that can watch thousands of triggers in one process, and the worker slot is freed for real work.

Edge cases

• The triggerer needs network access to whatever the trigger watches; a triggerer in a restricted subnet fails where workers succeeded.
• Triggerer HA: run multiple triggerers; deferred tasks are claimed by whichever triggerer picks them up.
• Not all providers support deferrable mode; check before refactoring a DAG around it.
• Deferred tasks still consume a DB row and scheduler attention; they are cheap, not free.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lDdhPk9-6f3WdKDe3c8eMQ