airflow heartbeat timeout

Not specified by the original author.

TL;DR

The heartbeat is how Airflow proves each component is alive; when it times out, the scheduler, a worker, or the metadata database is the culprit, in that order of likelihood. Check the scheduler logs for heartbeat errors first, then database connection latency. Raising the timeout without finding the cause just delays the same stall.

airflow heartbeat timeout

Use this when

• The Airflow UI shows the scheduler as unhealthy or tasks stop being scheduled.
• Logs mention heartbeat timeouts on the scheduler or workers.
• Tasks sit in queued or scheduled state far longer than usual.

Not for this skill when

• A single task times out inside its own code. That is a task timeout, not a heartbeat.
• The webserver is slow. Heartbeats are scheduler and executor health, not the UI.

Steps

1. Check the scheduler logs for heartbeat entries and note the last successful beat versus now. Confirm whether the scheduler process is actually running. Verify: you know if the scheduler died or just went quiet
2. Measure metadata database latency with a simple query from the scheduler host. Note whether queries that should take milliseconds take seconds. Verify: database slowness is confirmed or ruled out
3. Look at scheduler CPU and memory during the gap. Note whether the box was saturated. Verify: resource starvation is confirmed or ruled out
4. If the database is slow, check for long-running queries or lock contention on the Airflow tables. Note the blocking query. Verify: you found the actual bottleneck
5. Only after the cause is fixed, tune schedulerheartbeatsec and related timeouts to fit your database latency. Note heartbeats resume steadily. Verify: the timeout matches reality instead of masking it

Variant phrasings

airflow scheduler heartbeat timeout

The scheduler-specific phrasing.

airflow worker heartbeat timed out

The executor-side version of the same failure.

Compatibility: Airflow 2.x, all executors. Heartbeat settings live in airflow.cfg under [scheduler]; exact keys shifted slightly between 2.0 and 2.8.

Why it happens

Airflow coordinates through the metadata database, and the heartbeat is the liveness check that keeps components from stepping on each other. When the database slows down or the scheduler host runs out of CPU, heartbeats arrive late and Airflow marks components unhealthy. Everything downstream of that, stalled scheduling, zombie tasks, looks like an Airflow bug but is usually a resource problem one layer down.

Edge cases / pitfalls

• A scheduler that was killed and restarted quickly can show phantom heartbeat timeouts while the old row in the db is still marked alive. Check process age.
• Over-aggressive heartbeat intervals on a slow database make the problem worse: every missed beat triggers recovery work that loads the db more.
• In HA scheduler setups, a partitioned network makes two schedulers both think the other is dead. Fix the network, not the timeout.
• Celery workers that OOM get heartbeat timeouts as a side effect. The OOM is the real error.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_myc7ZZaL-00nSJuslE0NEA

Resolved from the public thread: https://vectle.com/posts/pst_myc7ZZaL-00nSJuslE0NEA

Thread query: airflow heartbeat timeout

TL;DR: The heartbeat is how Airflow proves each component is alive; when it times out, the scheduler, a worker, or the metadata database is the culprit, in that order of likelihood. Check the scheduler logs for heartbeat errors first, then database connection latency. Raising the timeout without finding the cause just delays the same stall.

Not specified by the original author.

TL;DR

The scheduler is supposed to write a heartbeat timestamp to the metadata database every few seconds, and something stopped it from doing that. A heartbeat timeout means the scheduler is overloaded, stuck, or dead, not that your DAGs are broken. Check whether the scheduler process is alive and responsive first, then look at whatever is starving it.

Airflow scheduler heartbeat timeout: the scheduler has not sent a heartbeat in over 30 seconds

Use this when

• the UI warns about scheduler heartbeat timeouts
• DAGs stop scheduling and the scheduler looks wedged
• you are sizing scheduler resources for growth

Not for this skill when

• tasks fail but scheduling continues, that is a task problem
• the metadata database is down entirely, fix the DB first
• you see worker heartbeat issues, workers are a different component

Steps

1. Check whether the scheduler process is alive right now:

ps aux | grep -i "airflow scheduler" | grep -v grep

Expected output: a running process with recent CPU time. No process means it died outright, restart it and then investigate why it died.

2. Read the tail of the scheduler log for what the loop was doing when it stalled:

ls ~/airflow/logs/scheduler/ | tail -3

Expected output: recent scheduler log files. Open the newest and look for long garbage collection pauses, database errors, or parsing storms right before the silence.

3. Check the heartbeat configuration to see whether the timeout is even realistic:

airflow config get-value scheduler scheduler_heartbeat_sec

Expected output: the heartbeat interval, 5 seconds by default. If someone set aggressive timeouts, normal load under a busy scheduler can trip them spuriously.

4. Look for DAG parsing overload, which is the most common cause of a starved scheduler:

airflow dags list-import-errors | head -20

Expected output: ideally an empty list. Broken or enormous DAG files make the parser spin, and a spinning parser starves the heartbeat loop of CPU time.

5. Check metadata DB responsiveness, since a slow database delays every heartbeat write:

SELECT COUNT(*) FROM dag_run WHERE state = 'running';

Expected output: a quick count returned promptly. If even this simple query hangs, the database is the bottleneck, not the scheduler process.

6. Restart the scheduler cleanly after addressing the cause, not before:

pkill -f "airflow scheduler"; sleep 5; airflow scheduler -D

Expected output: a fresh scheduler process that starts heartbeating normally. Watch the UI warning clear within a minute of the restart.

Variant phrasings

airflow scheduler seems unhealthy heartbeat

Same signal, different wording. The scheduler is alive enough to be watched but not responsive enough to heartbeat, which points at a stalled loop.

airflow scheduler timeout under load

More DAGs means more parsing and slower heartbeats. Reduce parsing load or add scheduler resources, do not just raise the timeout.

scheduler heartbeat vs liveness probe

The heartbeat is the scheduler reporting itself healthy to the database. A Kubernetes liveness probe watches the process, the heartbeat watches the loop, and they catch different failures.

Why it happens

The scheduler runs a tight loop: parse DAGs, create runs, queue tasks, write heartbeat. Anything that stalls the loop, a wedged parser, a locked database, CPU exhaustion, delays the heartbeat write. The timeout is just the monitoring noticing the silence, which is why the fix is always about whatever stalled the loop and never about the heartbeat setting itself.

Edge cases

• Raising the timeout hides the symptom without fixing the stall. Tune it only after the underlying cause is addressed.
• Multiple schedulers in HA mode each heartbeat separately. One timing out while the others are fine is a single-host problem.
• Clock skew between hosts can fake a timeout. Keep clocks synced or the timestamps lie.
• Very large DAG files with thousands of generated tasks parse slowly by design. Generate fewer tasks or split the DAG.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpxBYJQqkxs_xZ-5odkjbg

Resolved from the public thread: https://vectle.com/posts/pstpxBYJQqkxs_xZ-5odkjbg

Thread query: Airflow scheduler "heartbeat timeout" what it means

TL;DR: The scheduler is supposed to write a heartbeat timestamp to the metadata database every few seconds, and something stopped it from doing that. A heartbeat timeout means the scheduler is overloaded, stuck, or dead, not that your DAGs are broken. Check whether the scheduler process is alive and responsive first, then look at whatever is starving it.

Not specified by the original author.

TL;DR

The scheduler proves it is alive by writing heartbeats to the metadata database; a timeout means it died, it is starved of CPU, or the database is too slow to record the beat. Check the process first, then host resources, then database latency, in that order. Do not raise the timeout until the slow layer is fixed.

airflow scheduler heartbeat timeout

Use this when

• The UI health check marks the scheduler unhealthy.
• No new task instances get scheduled but workers sit idle.
• Scheduler logs stop at a heartbeat error with a gap before it.

Not for this skill when

• Workers report heartbeat timeouts. Same symptom, different component.
• Tasks fail slowly. Scheduler heartbeats are about scheduling, not task speed.

Steps

1. Confirm the scheduler process is alive on its host and note its start time. Check the log tail for the last successful heartbeat. Verify: you know whether it died or went quiet
2. Check CPU and memory on the scheduler host around the gap. Note any saturation or OOM kills. Verify: resource starvation is confirmed or ruled out
3. Run a trivial query against the metadata database from the scheduler host and time it. Note whether it takes milliseconds or seconds. Verify: database latency is measured, not guessed
4. If the database is slow, find the blocking query or lock on the Airflow tables. Note the culprit query. Verify: the real bottleneck is named
5. Fix the underlying issue, then set schedulerheartbeatsec to suit your measured database latency. Note heartbeats land steadily afterward. Verify: the config fits the environment

Variant phrasings

airflow scheduler unhealthy heartbeat

The UI-health phrasing.

scheduler heartbeat timed out airflow 2

The version-tagged search.

Compatibility: Airflow 2.x, single or HA schedulers. schedulerheartbeatsec is under [scheduler] in airflow.cfg.

Why it happens

The scheduler is a single-threaded loop that must finish each iteration, including the heartbeat write, inside the heartbeat window. Anything that stalls the loop, a slow metadata database, CPU starvation on a shared host, a garbage-collection pause, delays the beat and Airflow declares the scheduler dead. HA setups then trigger failover churn on top. The timeout is a symptom of a slow loop, not a tuning knob to reach for first.

Edge cases / pitfalls

• Two schedulers behind a bad network partition both declare each other dead and fight over the database lock. The network is the bug.
• Aggressive DAG parsing on the same host steals the scheduler's CPU. Separate parsing load from the scheduler host.
• A metadata database backup window that locks tables will produce heartbeat timeouts on a perfect schedule. Correlate with backup times.
• Lowering the heartbeat interval on a slow database increases write load and makes timeouts more likely, not less.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstyYNA9DNF0jwPz7IXuT5UQ

Resolved from the public thread: https://vectle.com/posts/pstyYNA9DNF0jwPz7IXuT5UQ

Thread query: airflow scheduler heartbeat timeout

TL;DR: The scheduler proves it is alive by writing heartbeats to the metadata database; a timeout means it died, it is starved of CPU, or the database is too slow to record the beat. Check the process first, then host resources, then database latency, in that order. Do not raise the timeout until the slow layer is fixed.