when agent tried to merge duplicate experiments it failed with a duplicate key error
Merge by moving runs, not by creating a combined experiment under a colliding name: a duplicate key error on merge means the target identity, experiment name or a run's unique fields, already exists. Use when an agent consolidating duplicate MLflow experiments. Not for two experiments that must stay separate for audit reasons, where the fix is cross-referencing rather than merging.
TL;DR
Merge by moving runs, not by creating a combined experiment under a colliding name: a duplicate key error on merge means the target identity, experiment name or a run's unique fields, already exists. Inventory both experiments' runs, decide per run whether it moves, is skipped as a true duplicate, or needs a renamed target, and perform the merge as run-level moves with the tracker API, verifying counts afterwards.
when agent tried to merge duplicate experiments it failed with a duplicate key errorUse this when
- The run or call fails with this symptom: agent tried to merge duplicate experiments and failed with a duplicate key error.
- You are working on an agent consolidating duplicate MLflow experiments and the failure repeats on the same run or configuration.
- You need the failing run, call, or stage isolated before changing code or thresholds.
Not for this skill when
- Two experiments that must stay separate for audit reasons, where the fix is cross-referencing rather than merging.
- The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.
Steps
- List both experiments and their runs, marking runs that exist in both by id and by content signature. Verify: the overlap is mapped
- Choose the surviving experiment and record why, including which one downstream links point at. Verify: the survivor is chosen explicitly
- Move runs in small batches with the API, skipping true duplicates deliberately rather than forcing them. Verify: moved and skipped counts are tracked
- Verify final counts and sample runs in the survivor, then delete or archive the emptied duplicate. Verify: the survivor holds the union and the duplicate is retired
Variant phrasings
agent merge duplicate experiments duplicate key error
The verbatim phrasing.
mlflow merge experiments agent move runs fix
The move-runs phrasing.
consolidate duplicate experiments agent key conflict
The consolidation phrasing.
Compatibility: Agent administration of MLflow experiments, merge operations
Why it happens
Experiments and runs have unique identities the store enforces, so a merge expressed as create-or-copy collides with anything already present. Real merging is a series of identity-aware moves, and the duplicate key error is the store refusing the shortcut.
Edge cases / pitfalls
- Dashboards and reports link to experiment ids; update the links or keep the linked experiment as the survivor.
- A run existing in both with different metrics is not a duplicate; it is a fork, and merging must choose content deliberately.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_OWE0688kpB31Md474-sY9w