Airflow 3.2.1 triggerer heartbeats but silently processes zero triggers

TL;DR

If deferred tasks hang in 3.2.1 while the triggerer appears healthy, do not trust the heartbeat: upgrade to 3.2.2 or later, where the lock-based serialization was replaced with response multiplexing so concurrent trigger requests no longer contend or deadlock.

Steps

1. If deferred tasks hang in 3.2.1 while the triggerer appears healthy, do not trust the heartbeat: upgrade to 3.2.2 or later, where the lock-based serialization was replaced with response multiplexing so concurrent trigger requests no longer contend or deadlock. As a diagnostic, check whether the number of processed triggers is actually advancing and whether deferred tasks start moving right after a triggerer restart; a triggerer that heartbeats but makes no progress matches this bug. Also keep custom logging config classes out of the triggerer (AIRFLOWLOGGINGLOGGINGCONFIGCLASS empty) since the reporter's custom logging handler interacted with the failure.

When to use

You are seeing this: Also keep custom logging config classes out of the triggerer (AIRFLOWLOGGINGLOGGINGCONFIGCLASS empty) since the reporter's custom logging handler interacted with the failure. Use this skill when you run into "Airflow 3.2.1 triggerer heartbeats but silently processes zero triggers".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 3.2.1, 3.2.2. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Also keep custom logging config classes out of the triggerer (AIRFLOWLOGGINGLOGGINGCONFIGCLASS empty) since the reporter's custom logging handler interacted with the failure.

Source: GitHub issue apache/airflow#65945 (closed, 11 comments): after upgrading to Airflow 3.2.1 the triggerer sent heartbeats and looked healthy to the scheduler, but silently processed zero triggers, so every deferred task stalled until it timed out. Root cause was a triggerer race condition and deadlock: triggers calling synchronous SDK methods could crash the triggerer's internal subprocess, and the partial fix in 3.2.1 introduced a new deadlock with the same visible symptom under load.

Verified resolution: If deferred tasks hang in 3.2.1 while the triggerer appears healthy, do not trust the heartbeat: upgrade to 3.2.2 or later, where the lock-based serialization was replaced with response multiplexing so concurrent trigger requests no longer contend or deadlock. As a diagnostic, check whether the number of processed triggers is actually advancing and whether deferred tasks start moving right after a triggerer restart; a triggerer that heartbeats but makes no progress matches this bug. Also keep custom logging config classes out of the triggerer (AIRFLOWLOGGINGLOGGINGCONFIGCLASS empty) since the reporter's custom logging handler interacted with the failure.

Source URL: https://github.com/apache/airflow/issues/65945