airflow scheduler heartbeat timeout

TL;DR

The scheduler proves it is alive by writing heartbeats to the metadata database; a timeout means it died, it is starved of CPU, or the database is too slow to record the beat. Check the process first, then host resources, then database latency, in that order. Do not raise the timeout until the slow layer is fixed.

airflow scheduler heartbeat timeout

Use this when

• The UI health check marks the scheduler unhealthy.
• No new task instances get scheduled but workers sit idle.
• Scheduler logs stop at a heartbeat error with a gap before it.

Not for this skill when

• Workers report heartbeat timeouts. Same symptom, different component.
• Tasks fail slowly. Scheduler heartbeats are about scheduling, not task speed.

Steps

1. Confirm the scheduler process is alive on its host and note its start time. Check the log tail for the last successful heartbeat. Verify: you know whether it died or went quiet
2. Check CPU and memory on the scheduler host around the gap. Note any saturation or OOM kills. Verify: resource starvation is confirmed or ruled out
3. Run a trivial query against the metadata database from the scheduler host and time it. Note whether it takes milliseconds or seconds. Verify: database latency is measured, not guessed
4. If the database is slow, find the blocking query or lock on the Airflow tables. Note the culprit query. Verify: the real bottleneck is named
5. Fix the underlying issue, then set schedulerheartbeatsec to suit your measured database latency. Note heartbeats land steadily afterward. Verify: the config fits the environment

Variant phrasings

airflow scheduler unhealthy heartbeat

The UI-health phrasing.

scheduler heartbeat timed out airflow 2

The version-tagged search.

Compatibility: Airflow 2.x, single or HA schedulers. schedulerheartbeatsec is under [scheduler] in airflow.cfg.

Why it happens

The scheduler is a single-threaded loop that must finish each iteration, including the heartbeat write, inside the heartbeat window. Anything that stalls the loop, a slow metadata database, CPU starvation on a shared host, a garbage-collection pause, delays the beat and Airflow declares the scheduler dead. HA setups then trigger failover churn on top. The timeout is a symptom of a slow loop, not a tuning knob to reach for first.

Edge cases / pitfalls

• Two schedulers behind a bad network partition both declare each other dead and fight over the database lock. The network is the bug.
• Aggressive DAG parsing on the same host steals the scheduler's CPU. Separate parsing load from the scheduler host.
• A metadata database backup window that locks tables will produce heartbeat timeouts on a perfect schedule. Correlate with backup times.
• Lowering the heartbeat interval on a slow database increases write load and makes timeouts more likely, not less.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstyYNA9DNF0jwPz7IXuT5UQ