airflow backfill created duplicate dag runs, how to fix

TL;DR

Duplicate runs from a backfill usually come from running the backfill twice, overlapping date ranges, or a backfill colliding with the regular schedule. Delete the extra runs, keeping one per logical date, then re-run the backfill with a single bounded date range and maxactiveruns set low. Verify one run per date before calling it fixed.

airflow backfill created duplicate dag runs, how to fix

Use this when

• A backfill produced two or more runs for the same logical date.
• Downstream tables show doubled rows after a backfill.
• The backfill overlapped with the normal schedule window.

Not for this skill when

• Scheduled runs duplicate without any backfill. Check schedule and catchup settings.
• Tasks within one run duplicated. That is a task retry or idempotency problem.

Steps

1. List DAG runs for the backfill window and identify duplicates: same logical date, multiple run ids. Note the duplicate pairs. Verify: the scope of duplication is mapped
2. Check whether the backfill was launched twice or overlapped the live schedule. Note the cause. Verify: the trigger is understood
3. Delete the extra runs, keeping the one per logical date whose tasks actually completed. Note one run remains per date. Verify: the run list is clean
4. Clean up any doubled downstream writes from the duplicate runs, or mark them for reprocessing. Note downstream tables are consistent. Verify: the data matches the runs
5. Re-run the backfill if needed with one bounded date range, maxactiveruns low, and the regular schedule paused for the window. Note exactly one run per date is created. Verify: the rerun is duplicate-free

Variant phrasings

airflow backfill duplicate runs

The compact phrasing.

airflow backfill overlapping runs

The cause-shaped search.

Compatibility: Airflow 2.x CLI backfill and UI-triggered backfills. Run deletion via the UI or API works on all 2.x.

Why it happens

Backfills create runs per logical date, and Airflow does not deduplicate against runs created by another backfill invocation or the live schedule for the same dates. Running the command twice, or backfilling a window the scheduler is also covering, produces two runs per date that both write downstream. The duplicates are real runs doing real work, which is why downstream tables double up. Prevention is procedural: one backfill, one bounded range, schedule paused.

Edge cases / pitfalls

• Deleting a run does not undo its downstream writes. The data cleanup is a separate step people forget.
• Tasks without idempotency double-apply on duplicate runs. Backfill safety starts with idempotent tasks.
• Pausing the DAG pauses the schedule but a backfill still creates runs. Pause-then-backfill is the safe pattern for overlapping windows.
• The --rerun-failed-tasks flag on a second backfill can resurrect runs you already deleted. Check run state before re-backfilling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AsPoff1ScNw7HXtimYr3dg