## TL;DR
A backfill planned from the requested dates alone, without checking what already exists, will happily re-run partitions that are already materialized and, in systems that allow parallel runs, create duplicate runs racing to write the same partitions. Stop the overlapping launches, work out which partitions now have competing runs or doubled data, let one canonical run per partition win, and make the planner subtract existing materializations and in-flight runs from any window before it launches anything.

```text
agent's backfill window overlapped existing partitions, duplicate runs created
```

## Use this when
- A backfill creates runs for dates that were already loaded and healthy.
- Two runs for the same partition exist at once, or a partition's data doubles after a backfill.
- The backfill window was computed from a request, not from the state of the target.

## Not for this skill when
- The overlap is an intentional re-backfill after a logic fix, planned to replace the old partitions. That needs partition replacement discipline, not cancellation.
- The duplicates are historical runs in the log that never wrote data. History alone is untidy, not harmful.

## Steps
1. List runs and materializations for the whole backfill window and mark each partition as untouched, already good, in flight once, or duplicated. Note the partition level map. Verify: the overlap is visible partition by partition
2. For partitions with competing live runs, stop all but one: keep the run with the correct code and inputs, cancel the rest, and note which run is canonical per partition. Verify: no partition has two writers racing
3. Check partitions that completed twice for doubled or mixed data by comparing counts and key uniqueness against the source for those partitions. Note the partitions needing a rebuild. Verify: data damage is separated from mere run duplication
4. Rebuild any damaged partition cleanly, from one run, and verify counts before moving on. Note the rebuilt set. Verify: every partition ends with exactly one load's worth of data
5. Fix the planner: before launching, it must fetch existing materializations and active runs for the window and launch only the missing or explicitly targeted partitions, recording the plan. Dry run the original request and note it now plans only the gap. Verify: the same request can no longer create the overlap

## Variant phrasings
### backfill overlapped partitions duplicate dagster airflow runs planner check existing
The planner check phrasing.
### backfill created duplicate runs for already materialized partitions cleanup
The cleanup shaped search.

Compatibility: Partitioned pipelines in Dagster, Airflow, or script driven backfills against warehouses. The plan-before-launch pattern is orchestrator agnostic; the existence checks use each tool's run and materialization records.

## Why it happens
Backfill requests are expressed in the language of need, the dates someone wants, while the system state lives in partitions and runs. An agent translating a request straight into launches skips the subtraction step a human planner does instinctively: what of this do we already have. Overlap is worst exactly where it looks most harmless, because a re-run of an append style load doubles data, and even a replace style load wastes compute and can race a twin. Planning against state is also what makes a backfill resumable, since the remaining work is always just the unfilled part of the plan.

## Edge cases / pitfalls
- A partition that exists but was built by buggy code should be in the plan on purpose. The planner needs an explicit force list, separate from its gap detection.
- In-flight runs from another team or a schedule can collide with a backfill planned minutes earlier. Recheck state at launch time, not only at plan time.
- Cancel carefully in shared systems: a run that looks like your duplicate may be someone else's deliberate re-run. Coordinate before cancelling what you did not launch.
- Keep the plan and the outcome together. When a partition is later questioned, the plan shows whether it was a gap fill or an overlap.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4qv27C0OH9sy8dkWhBPthQ