airflow deferrable sensor stuck in deferred state, triggerer not picking up task

TL;DR

A deferred task is not waiting on the worker at all. It handed a trigger object to the triggerer service, which polls cheaply until the condition is met, then wakes the task. Stuck in deferred almost always means the triggerer process is not running, cannot import the trigger class, or has hit its capacity. Check triggerer health and its logs for the trigger id before touching the sensor itself.

airflow deferrable sensor stuck in deferred state, triggerer not picking up task

Use this when

• A deferrable sensor or operator sits in deferred far past the expected wait.
• Many unrelated deferred tasks across DAGs are stuck at the same time.
• The task log shows it deferred successfully and then nothing happens.

Not for this skill when

• A poke mode sensor holds a worker slot and times out. That sensor never deferred.
• The external condition genuinely has not happened yet. Check the source system first.

Steps

1. Confirm the triggerer service is running: check the process or deployment and its heartbeat in the Airflow UI or the trigger table. Note whether any triggerer has checked in recently. Verify: triggerer liveness is known
2. Read the triggerer logs for the trigger id tied to your task instance and note import errors or repeated exceptions for that trigger class. Note the exact error. Verify: a broken trigger class is confirmed or ruled out
3. Check trigger capacity: how many triggers are registered versus the triggerer capacity setting, and whether one noisy DAG registered thousands. Note the counts. Verify: capacity exhaustion is confirmed or ruled out
4. If the trigger class fails to import on the triggerer, install the same provider package there that the workers use, and restart the triggerer. Note the trigger resumes logging. Verify: the triggerer runs the same code as the workers
5. If the trigger row is corrupt or wedged, clear the task instance so it re-defers with a fresh trigger, then watch it leave deferred when the condition lands. Note the wake-up in the task log. Verify: the task resumes and completes

Variant phrasings

airflow deferred task never resumes triggerer down

The triggerer-down phrasing.

airflow deferrable operator stuck deferred

The operator version of the same stall.

Compatibility: Airflow 2.2 and later with deferrable operators and a running triggerer; Airflow 3 keeps the same model. Trigger capacity defaults differ by version, so check your own configuration value.

Why it happens

Deferrable operators exist to stop sensors burning worker slots while waiting. The task serializes a small trigger object, exits, and the triggerer, a separate asyncio service, polls many triggers in one process. That design moves the failure surface: the worker and scheduler can be perfectly healthy while every deferred task in the system is frozen because the one component nobody monitors, the triggerer, is down, misconfigured, or missing a provider package its triggers need.

Edge cases / pitfalls

• One triggerer is a single point of failure for waits. Run at least two in production; they share work through the trigger table.
• A trigger that raises immediately on every poll spams the triggerer log and can starve other triggers. Fix the trigger code, not the sensor timeout.
• Clearing a deferred task cancels its trigger. If the external job the trigger watched keeps running, expect to reconcile it separately.
• Custom triggers must be importable from a path the triggerer sees. Code that lives only in the DAG folder packaging of workers may not be there.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_J7pxVQtoulhrafa92kAwhw