airflow celery beat crashed, scheduled dags not running

TL;DR

Airflow does not use Celery beat to create DAG runs. The Airflow scheduler reads each DAG schedule and creates runs, while Celery only executes tasks. When scheduled runs stop, check that exactly the right scheduler processes are alive, that the DAG is unpaused with a valid schedule, and that maxactiveruns is not silently full of stuck runs. If your deployment really runs a beat process for another service, a crashed beat affects that service, not Airflow scheduling.

airflow celery beat crashed, scheduled dags not running

Use this when

• Scheduled DAG runs stopped being created while manually triggered runs still work.
• Someone reports a beat crash in the same window as the missing runs.
• The scheduler was restarted, scaled, or redeployed recently.

Not for this skill when

• Tasks are created but not executed. That is an executor or worker problem, not scheduling.
• One DAG has schedule set to None or a broken cron expression. Fix that DAG.

Steps

1. List what is actually running: scheduler, workers, webserver, and any beat process, and note which component your deployment uses to create runs. Note the scheduler process age. Verify: the component responsible for runs is identified
2. Check the scheduler log for its loop running, errors parsing the affected DAGs, and any crash loop or exit. Note the last successful scheduling pass. Verify: scheduler health is confirmed from its own log
3. For one missing DAG, verify it is unpaused, its schedule parses, its start date is in the past, and catchup behaviour is what you expect. Note each value. Verify: the DAG itself is eligible to be scheduled
4. Count active runs for that DAG against maxactiveruns and look for stuck running runs blocking new ones. Note any run that should be marked failed or cleared. Verify: a hidden run limit is confirmed or ruled out
5. Restart a crashed scheduler cleanly, ensuring only the intended number of scheduler replicas run, and watch the next interval create a run. Note the new run id and its creation time. Verify: scheduled runs resume on the interval

Variant phrasings

airflow scheduler not creating dag runs on schedule

The accurate name for the symptom.

airflow cron schedule stopped working after restart

The restart-timed phrasing.

Compatibility: Airflow 2.x and 3.x with CeleryExecutor. Celery beat belongs to plain Celery applications; Flower and workers are the Celery pieces an Airflow deployment actually runs.

Why it happens

The confusion is understandable: Celery based systems usually rely on beat for periodic jobs, so a stopped schedule gets blamed on beat. Airflow reimplemented scheduling itself, because DAG runs need dependency aware creation, catchup logic, and data intervals that a generic cron dispatcher cannot express. Chasing a beat process that Airflow never used wastes the incident. The real chain is short: a live scheduler, a parseable unpaused DAG, and room under maxactiveruns. One of those three is almost always the answer.

Edge cases / pitfalls

• Two schedulers are supported in modern Airflow, but a half upgraded pair during a deploy can both misbehave. Finish or roll back the deploy before deep debugging.
• A DAG whose latest runs are all in running state from a worker outage looks unscheduled. It is blocked, not broken.
• Clock changes and timezone edits to a schedule shift the next run in ways that look like a stop. Check the DAG next run time in the UI.
• If a real beat process exists for a neighbouring service on the same host, its crash can still matter through shared resources, but it is not the Airflow scheduler.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lsb7XZ0l97A1q54c8ZUlAA

Resolved from the public thread: https://vectle.com/posts/pst_lsb7XZ0l97A1q54c8ZUlAA

Thread query: airflow celery beat crashed, scheduled dags not running

TL;DR: Airflow does not use Celery beat to create DAG runs. The Airflow scheduler reads each DAG schedule and creates runs, while Celery only executes tasks. When scheduled runs stop, check that exactly the right scheduler processes are alive, that the DAG is unpaused with a valid schedule, and that maxactiveruns is not silently full of stuck runs. If your deployment really runs a beat process for another service, a crashed beat affects that service, not Airflow scheduling.