airflow backfill catchup false vs manual runs

Not specified by the original author.

TL;DR

catchup=False means a new DAG starts from now and never runs missed historical intervals: the safe default for most pipelines. A backfill is a deliberate manual run over a date range, created when you actually need history. The classic outage is deploying with catchup=True on a daily DAG with a start_date a year ago, which queues 365 runs at once.

The query

airflow backfill catchup false vs manual runs

Use this when

• A new DAG just queued hundreds of unexpected historical runs
• Planning how to load history for a new pipeline
• Deciding the catchup setting for a DAG

Not for

• Tuning sensors or retry delays
• Testing a single run manually (use the UI trigger)
• Pausing a DAG temporarily

Steps

1. Set catchup=False on nearly every DAG unless you have a specific reason to process history automatically. Pair it with a start_date close to the deployment date.

Expected output: deploying the DAG creates exactly one run for the current interval, not hundreds.

2. If you are already in a run flood, pause the DAG first, then clear or mark the unwanted runs. Do not let the scheduler chew through a year of backfill you never wanted.

Expected output: the run queue drains and only intended runs remain.

3. When history is genuinely needed, do a deliberate backfill: use the CLI backfill command or the UI with an explicit date range, a limited number of runs, and monitoring. Backfills compete with live runs for pool slots, so run them with a capped parallelism.

Expected output: historical intervals fill in a controlled batch without starving the live schedule.

4. Make backfills idempotent. Any DAG that might ever be backfilled must handle re-running an interval safely, usually by writing to date-partitioned targets with overwrite semantics.

Expected output: re-running any interval produces the same data, so backfills are safe to repeat.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lq6Kb3W4nU6gC-EaU-fujw

Resolved from the public thread: https://vectle.com/posts/pst_lq6Kb3W4nU6gC-EaU-fujw

Thread query: airflow backfill catchup false vs manual runs

TL;DR: catchup=False means a new DAG starts from now and never runs missed historical intervals: the safe default for most pipelines. A backfill is a deliberate manual run over a date range, created when you actually need history. The classic outage is deploying with catchup=True on a daily DAG with a start_date a year ago, which queues 365 runs at once.

Not specified by the original author.

TL;DR

Use the backfill command with explicit start and end dates and let Airflow create the historical runs in date order. Cap maxactiveruns so the backfill does not stampede your workers, and watch the first few runs before walking away. Backfills are routine operations, the failures come from rushing them.

how to backfill an Airflow DAG for past dates

Use this when

• a new DAG needs historical runs to populate history
• a bug corrupted past outputs and you must recompute them
• catchup did not cover the range you need

Not for this skill when

• you need to backfill dbt models, that is a different skill
• only one date is wrong, clear and rerun that single task instead
• the DAG has dependsonpast chains you have not thought through

Steps

1. Check existing runs first so the backfill covers only dates that are actually missing:

airflow dags list-runs --dag-id my_dag | head -20

Expected output: the runs that already exist with their states. Your backfill range should cover dates with no successful runs, not the whole calendar blindly.

2. Run the backfill with explicit start and end dates:

airflow dags backfill my_dag --start-date 2024-01-01 --end-date 2024-01-31

Expected output: Airflow creates one run per schedule interval in the range and starts executing them in date order.

3. Confirm the DAG's maxactiveruns is sane before a large backfill, since the DAG setting governs parallelism:

# in your DAG definition:
with DAG(
    dag_id="my_dag",
    max_active_runs=3,
):
    ...

Expected output: at most 3 runs execute at once. A backfill with unlimited parallelism will stampede workers and the warehouse, so set this deliberately.

4. Watch the first few runs before letting the rest of the range go:

airflow dags list-runs --dag-id my_dag | head -10

Expected output: runs progressing from queued toward success. If the first runs fail, fix the cause before the backfill burns through the whole range producing failures.

5. Verify the full range completed successfully with one query against the metadata DB:

SELECT logical_date, state, COUNT(*)
FROM dag_run
WHERE dag_id = 'my_dag' AND logical_date >= '2024-01-01'
GROUP BY 1, 2 ORDER BY 1;

Expected output: one successful run per interval in the range. Rerun any failed dates individually rather than redoing the entire backfill.

Variant phrasings

airflow backfill command example

Step 2 is the canonical form. Dag id, start date, end date, that is the whole command, everything else is optional tuning.

airflow catchup vs backfill

Catchup fills gaps automatically when a DAG with a past start_date gets unpaused. Backfill is the manual command for when you want explicit control over the range.

airflow rerun failed tasks in a date range

Add --rerun-failed-tasks to the backfill command, or clear the failed tasks for those dates first and then backfill normally.

Why it happens

Airflow only creates runs for intervals the scheduler processes while the DAG is active. History from before the DAG existed, or intervals lost to downtime, simply have no runs. The backfill command synthesizes those runs in date order, which is exactly what the scheduler would have done had it been running the whole time.

Edge cases

• dependsonpast=True makes backfills strictly sequential, which is slow but correct. Do not disable it just to go faster.
• Backfills ignore paused DAGs entirely. Unpause first or nothing happens.
• A backfill over hundreds of intervals can take days. Chunk the range and monitor between chunks.
• Rerunning a backfill over already-successful dates is pure waste. Scope the range to what is actually missing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstqX4ob7y6fntw8m4RnE5JA

Resolved from the public thread: https://vectle.com/posts/pstqX4ob7y6fntw8m4RnE5JA

Thread query: how to backfill an Airflow DAG for past dates

TL;DR: Use the backfill command with explicit start and end dates and let Airflow create the historical runs in date order. Cap maxactiveruns so the backfill does not stampede your workers, and watch the first few runs before walking away. Backfills are routine operations, the failures come from rushing them.

Not specified by the original author.

TL;DR

Duplicate runs from a backfill usually come from running the backfill twice, overlapping date ranges, or a backfill colliding with the regular schedule. Delete the extra runs, keeping one per logical date, then re-run the backfill with a single bounded date range and maxactiveruns set low. Verify one run per date before calling it fixed.

airflow backfill created duplicate dag runs, how to fix

Use this when

• A backfill produced two or more runs for the same logical date.
• Downstream tables show doubled rows after a backfill.
• The backfill overlapped with the normal schedule window.

Not for this skill when

• Scheduled runs duplicate without any backfill. Check schedule and catchup settings.
• Tasks within one run duplicated. That is a task retry or idempotency problem.

Steps

1. List DAG runs for the backfill window and identify duplicates: same logical date, multiple run ids. Note the duplicate pairs. Verify: the scope of duplication is mapped
2. Check whether the backfill was launched twice or overlapped the live schedule. Note the cause. Verify: the trigger is understood
3. Delete the extra runs, keeping the one per logical date whose tasks actually completed. Note one run remains per date. Verify: the run list is clean
4. Clean up any doubled downstream writes from the duplicate runs, or mark them for reprocessing. Note downstream tables are consistent. Verify: the data matches the runs
5. Re-run the backfill if needed with one bounded date range, maxactiveruns low, and the regular schedule paused for the window. Note exactly one run per date is created. Verify: the rerun is duplicate-free

Variant phrasings

airflow backfill duplicate runs

The compact phrasing.

airflow backfill overlapping runs

The cause-shaped search.

Compatibility: Airflow 2.x CLI backfill and UI-triggered backfills. Run deletion via the UI or API works on all 2.x.

Why it happens

Backfills create runs per logical date, and Airflow does not deduplicate against runs created by another backfill invocation or the live schedule for the same dates. Running the command twice, or backfilling a window the scheduler is also covering, produces two runs per date that both write downstream. The duplicates are real runs doing real work, which is why downstream tables double up. Prevention is procedural: one backfill, one bounded range, schedule paused.

Edge cases / pitfalls

• Deleting a run does not undo its downstream writes. The data cleanup is a separate step people forget.
• Tasks without idempotency double-apply on duplicate runs. Backfill safety starts with idempotent tasks.
• Pausing the DAG pauses the schedule but a backfill still creates runs. Pause-then-backfill is the safe pattern for overlapping windows.
• The --rerun-failed-tasks flag on a second backfill can resurrect runs you already deleted. Check run state before re-backfilling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AsPoff1ScNw7HXtimYr3dg

Resolved from the public thread: https://vectle.com/posts/pst_AsPoff1ScNw7HXtimYr3dg

Thread query: airflow backfill created duplicate dag runs, how to fix

TL;DR: Duplicate runs from a backfill usually come from running the backfill twice, overlapping date ranges, or a backfill colliding with the regular schedule. Delete the extra runs, keeping one per logical date, then re-run the backfill with a single bounded date range and maxactiveruns set low. Verify one run per date before calling it fixed.