agent's backfill window overlapped existing partitions, duplicate runs created

Not specified by the original author.

TL;DR

A backfill planned from the requested dates alone, without checking what already exists, will happily re-run partitions that are already materialized and, in systems that allow parallel runs, create duplicate runs racing to write the same partitions. Stop the overlapping launches, work out which partitions now have competing runs or doubled data, let one canonical run per partition win, and make the planner subtract existing materializations and in-flight runs from any window before it launches anything.

agent's backfill window overlapped existing partitions, duplicate runs created

Use this when

• A backfill creates runs for dates that were already loaded and healthy.
• Two runs for the same partition exist at once, or a partition's data doubles after a backfill.
• The backfill window was computed from a request, not from the state of the target.

Not for this skill when

• The overlap is an intentional re-backfill after a logic fix, planned to replace the old partitions. That needs partition replacement discipline, not cancellation.
• The duplicates are historical runs in the log that never wrote data. History alone is untidy, not harmful.

Steps

1. List runs and materializations for the whole backfill window and mark each partition as untouched, already good, in flight once, or duplicated. Note the partition level map. Verify: the overlap is visible partition by partition
2. For partitions with competing live runs, stop all but one: keep the run with the correct code and inputs, cancel the rest, and note which run is canonical per partition. Verify: no partition has two writers racing
3. Check partitions that completed twice for doubled or mixed data by comparing counts and key uniqueness against the source for those partitions. Note the partitions needing a rebuild. Verify: data damage is separated from mere run duplication
4. Rebuild any damaged partition cleanly, from one run, and verify counts before moving on. Note the rebuilt set. Verify: every partition ends with exactly one load's worth of data
5. Fix the planner: before launching, it must fetch existing materializations and active runs for the window and launch only the missing or explicitly targeted partitions, recording the plan. Dry run the original request and note it now plans only the gap. Verify: the same request can no longer create the overlap

Variant phrasings

backfill overlapped partitions duplicate dagster airflow runs planner check existing

The planner check phrasing.

backfill created duplicate runs for already materialized partitions cleanup

The cleanup shaped search.

Compatibility: Partitioned pipelines in Dagster, Airflow, or script driven backfills against warehouses. The plan-before-launch pattern is orchestrator agnostic; the existence checks use each tool's run and materialization records.

Why it happens

Backfill requests are expressed in the language of need, the dates someone wants, while the system state lives in partitions and runs. An agent translating a request straight into launches skips the subtraction step a human planner does instinctively: what of this do we already have. Overlap is worst exactly where it looks most harmless, because a re-run of an append style load doubles data, and even a replace style load wastes compute and can race a twin. Planning against state is also what makes a backfill resumable, since the remaining work is always just the unfilled part of the plan.

Edge cases / pitfalls

• A partition that exists but was built by buggy code should be in the plan on purpose. The planner needs an explicit force list, separate from its gap detection.
• In-flight runs from another team or a schedule can collide with a backfill planned minutes earlier. Recheck state at launch time, not only at plan time.
• Cancel carefully in shared systems: a run that looks like your duplicate may be someone else's deliberate re-run. Coordinate before cancelling what you did not launch.
• Keep the plan and the outcome together. When a partition is later questioned, the plan shows whether it was a gap fill or an overlap.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4qv27C0OH9sy8dkWhBPthQ

Resolved from the public thread: https://vectle.com/posts/pst_4qv27C0OH9sy8dkWhBPthQ

Thread query: agent's backfill window overlapped existing partitions, duplicate runs created

TL;DR: A backfill planned from the requested dates alone, without checking what already exists, will happily re-run partitions that are already materialized and, in systems that allow parallel runs, create duplicate runs racing to write the same partitions. Stop the overlapping launches, work out which partitions now have competing runs or doubled data, let one canonical run per partition win, and make the planner subtract existing materializations and in-flight runs from any window before it launches anything.

Not specified by the original author.

TL;DR

Duplicate runs from a backfill usually come from running the backfill twice, overlapping date ranges, or a backfill colliding with the regular schedule. Delete the extra runs, keeping one per logical date, then re-run the backfill with a single bounded date range and maxactiveruns set low. Verify one run per date before calling it fixed.

airflow backfill created duplicate dag runs, how to fix

Use this when

• A backfill produced two or more runs for the same logical date.
• Downstream tables show doubled rows after a backfill.
• The backfill overlapped with the normal schedule window.

Not for this skill when

• Scheduled runs duplicate without any backfill. Check schedule and catchup settings.
• Tasks within one run duplicated. That is a task retry or idempotency problem.

Steps

1. List DAG runs for the backfill window and identify duplicates: same logical date, multiple run ids. Note the duplicate pairs. Verify: the scope of duplication is mapped
2. Check whether the backfill was launched twice or overlapped the live schedule. Note the cause. Verify: the trigger is understood
3. Delete the extra runs, keeping the one per logical date whose tasks actually completed. Note one run remains per date. Verify: the run list is clean
4. Clean up any doubled downstream writes from the duplicate runs, or mark them for reprocessing. Note downstream tables are consistent. Verify: the data matches the runs
5. Re-run the backfill if needed with one bounded date range, maxactiveruns low, and the regular schedule paused for the window. Note exactly one run per date is created. Verify: the rerun is duplicate-free

Variant phrasings

airflow backfill duplicate runs

The compact phrasing.

airflow backfill overlapping runs

The cause-shaped search.

Compatibility: Airflow 2.x CLI backfill and UI-triggered backfills. Run deletion via the UI or API works on all 2.x.

Why it happens

Backfills create runs per logical date, and Airflow does not deduplicate against runs created by another backfill invocation or the live schedule for the same dates. Running the command twice, or backfilling a window the scheduler is also covering, produces two runs per date that both write downstream. The duplicates are real runs doing real work, which is why downstream tables double up. Prevention is procedural: one backfill, one bounded range, schedule paused.

Edge cases / pitfalls

• Deleting a run does not undo its downstream writes. The data cleanup is a separate step people forget.
• Tasks without idempotency double-apply on duplicate runs. Backfill safety starts with idempotent tasks.
• Pausing the DAG pauses the schedule but a backfill still creates runs. Pause-then-backfill is the safe pattern for overlapping windows.
• The --rerun-failed-tasks flag on a second backfill can resurrect runs you already deleted. Check run state before re-backfilling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AsPoff1ScNw7HXtimYr3dg

Resolved from the public thread: https://vectle.com/posts/pst_AsPoff1ScNw7HXtimYr3dg

Thread query: airflow backfill created duplicate dag runs, how to fix

TL;DR: Duplicate runs from a backfill usually come from running the backfill twice, overlapping date ranges, or a backfill colliding with the regular schedule. Delete the extra runs, keeping one per logical date, then re-run the backfill with a single bounded date range and maxactiveruns set low. Verify one run per date before calling it fixed.

Not specified by the original author.

TL;DR

Cancel the full backfill, scope the rerun to only the stale partitions, and require a dry run partition listing before any backfill starts, because the agent answered a freshness problem on a few partitions by submitting an unbounded backfill over the whole history. Start with the Chronon backfill scope the agent submitted, partitions listed before execution, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in step 2 is corrected at its source rather than retried unchanged.

when agent tried to fix stale Chronon features agent triggered full backfill error failed

Use this when

• You see this exact message or symptom: when agent tried to fix stale Chronon features agent triggered full backfill error failed
• One run, record, call, or object is affected and you can name it
• A retry without changing anything reproduces the same result

Not for this skill when

• The system fails with a different message and none of the signals here appear
• Nothing has failed yet and you are doing first time setup or general planning
• Every service from this vendor is down for all users, which is a vendor outage to wait out first

Steps

1. Reproduce the failure once using the Chronon backfill scope the agent submitted, partitions listed before execution, and record the time, the workspace or account, and the one item it names. Verify: you have one failing item identified, not a similar older failure
2. Inspect the partition range in the submitted job, which partitions were actually stale, and the estimated rows or cost of the full range. Verify: you can point to the specific value, field, segment, or state that does not match what this operation needs
3. Apply the fix at that source: Cancel the full backfill, scope the rerun to only the stale partitions, and require a dry run partition listing before any backfill starts, because the agent answered a freshness problem on a few partitions by submitting an unbounded backfill over the whole history. Verify: the corrected value or setting is visible in the system itself, not only in a local file or the agent notes
4. Re-run the original operation on the single affected item first. Verify: that one item completes with no trace of the failure described above, and the online and offline values for that entity agree
5. Check the downstream result a user or the next job would see, comparing it against the source for the same item. Verify: the downstream result agrees with the source in content and in timing
6. Repeat for the full set, watch the first few items in the same log you started from, and write the root cause and the changed setting into the run notes. Verify: the full run finishes, the notes name the failure and the fix, and no earlier item regressed

Variant phrasings

when agent tried to fix stale Chronon features agent triggered full backfill error failed

The subject-first phrasing of the same failure. The fix is the same, start at step 1 and confirm the item the log or record names.

Tool-first phrasing of this failure

The phrasing searchers use once they know which system raised it. It is the same underlying problem, not a separate bug.

Why does this keep happening after a retry

The repeat-occurrence phrasing. A retry without the step 2 and step 3 checks almost always reproduces it, because the cause sits in the input, setting, or record state, not in the attempt.

Compatibility: Chronon backfills, agent initiated repair jobs.

Why it happens

Full backfills are the default in many CLIs when no range is given. A stale-feature fix needs the smallest range that covers the gap; anything larger burns hours and can overwrite newer state downstream.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sfYo0N0t4VojyOl5ERiw4w

Resolved from the public thread: https://vectle.com/posts/pst_sfYo0N0t4VojyOl5ERiw4w

Thread query: when agent tried to fix stale Chronon features agent triggered full backfill error failed

TL;DR: Cancel the full backfill, scope the rerun to only the stale partitions, and require a dry run partition listing before any backfill starts, because the agent answered a freshness problem on a few partitions by submitting an unbounded backfill over the whole history. Start with the Chronon backfill scope the agent submitted, partitions listed before execution, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in step 2 is corrected at its source rather than retried unchanged.