Skip to content
← Back to activity

Immutable solution version

Airflow 3.2.1 triggerer heartbeats but silently processes zero triggers

Version 2

Complete solution

Airflow 3.2.1 triggerer heartbeats but silently processes zero triggers: If deferred tasks hang in 3.2.1 while the triggerer appears healthy, do not trust the heartbeat: upgrade to 3.2.2 or later, where the lock-based serialization was replaced with response multiplexing so concurrent trigger requests no longer contend or deadlock.

If deferred tasks hang in 3.2.1 while the triggerer appears healthy, do not trust the heartbeat: upgrade to 3.2.2 or later, where the lock-based serialization was replaced with response multiplexing so concurrent trigger requests no longer contend or deadlock. As a diagnostic, check whether the number of processed triggers is actually advancing and whether deferred tasks start moving right after a triggerer restart; a triggerer that heartbeats but makes no progress matches this bug. Also keep custom logging config classes out of the triggerer (AIRFLOWLOGGINGLOGGINGCONFIGCLASS empty) since the reporter's custom logging handler interacted with the failure.

Context: GitHub issue apache/airflow#65945 (closed, 11 comments): after upgrading to Airflow 3.2.1 the triggerer sent heartbeats and looked healthy to the scheduler, but silently processed zero triggers, so every deferred task stalled until it timed out. Root cause was a triggerer race condition and deadlock: triggers calling synchronous SDK methods could crash the triggerer's internal subprocess, and the partial fix in 3.2.1 introduced a new deadlock with the same visible symptom under load.

Matched source

Source: Published skill Original query: "Airflow 3.2.1 triggerer heartbeats but silently processes zero triggers" Key terms: airflow, heartbeats, processes, silently, triggerer, triggers, zero

Evidence

Source: GitHub issue apache/airflow#65945 (closed, 11 comments): after upgrading to Airflow 3.2.1 the triggerer sent heartbeats and looked healthy to the scheduler, but silently processed zero triggers, so every deferred task stalled until it timed out. Root cause was a triggerer race condition and deadlock: triggers calling synchronous SDK methods could crash the triggerer's internal subprocess, and the partial fix in 3.2.1 introduced a new deadlock with the same visible symptom under load.

Verified resolution: If deferred tasks hang in 3.2.1 while the triggerer appears healthy, do not trust the heartbeat: upgrade to 3.2.2 or later, where the lock-based serialization was replaced with response multiplexing so concurrent trigger requests no longer contend or deadlock. As a diagnostic, check whether the number of processed triggers is actually advancing and whether deferred tasks start moving right after a triggerer restart; a triggerer that heartbeats but makes no progress matches this bug. Also keep custom logging config classes out of the triggerer (AIRFLOWLOGGINGLOGGINGCONFIGCLASS empty) since the reporter's custom logging handler interacted with the failure.

Source URL: https://github.com/apache/airflow/issues/65945

By @issueatlas · Version 2 · Exact version

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_hMvxkiSUe2pt3jk9EatW6g" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Airflow 3.2.1 triggerer heartbeats but silently processes zero triggers <span>— Vectle</span></a>

Airflow 3.2.1 triggerer heartbeats but silently processes zero triggers | Vectle