experiment metrics logged twice after training job retry duplicates run history error

TL;DR

Make the retry resume the same run instead of opening a new logging session: when a training job retries, it re-logs epochs the tracker already has, so key the tracker run to the job attempt and resume it, or discard the partial duplicate before reporting results. Retries are normal in training infrastructure. Trackers that treat every attempt as a new run, or that append without checking, turn one training run into a doubled history.

experiment metrics logged twice after training job retry duplicates run history error

Use this when

• The check or run fails with this symptom: experiment metrics are logged twice after a training job retry and duplicate the run history.
• You are working on a tracked training job running on retrying infrastructure such as a scheduler or a spot instance and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• Two genuinely separate runs with identical parameters, which should both exist and be distinguished by their run metadata.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Compare the duplicated run records: timestamps, attempt identifiers, and step ranges. Verify: the duplicates share one job and overlapping steps
2. Check how the job creates its tracker run on start. Verify: each attempt creates or resumes without checking for an existing run
3. Pass a stable run identifier from the job, and resume that run on retry, or end the failed attempt run as failed and start reporting from the successful one only. Verify: a retry produces one coherent history
4. Clean the existing duplicate by removing the partial attempt record. Verify: reports show a single run
5. Add attempt number and job id as run tags. Verify: future duplicates are recognisable at a glance

Variant phrasings

experiment metrics logged twice training job retry duplicates

The shorter phrasing focused on the named failure.

duplicate run history after retry error

The symptom phrasing a searcher tries next.

training retry duplicate metrics fix

The variant naming the failing stage or tool directly.

Compatibility: MLflow, W&B, and similar trackers with run resume, retrying training schedulers

Why it happens

Training frameworks log per epoch with no knowledge of job attempts, and trackers accept the writes at face value. A retry after a preemption replays the early epochs into the same or a sibling run, and every downstream average, chart, and best-run selection inherits the duplication.

Edge cases / pitfalls

• Deduplicating by step value is unsafe; identical losses can occur legitimately. Dedupe by attempt identity.
• A resumed run must not re-log parameters; most trackers reject or duplicate them, compounding the original problem.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstj3iTCI71kM--Fqggesm9w

Resolved from the public thread: https://vectle.com/posts/pstj3iTCI71kM--Fqggesm9w

Thread query: experiment metrics logged twice after training job retry duplicates run history error

TL;DR: Make the retry resume the same run instead of opening a new logging session: when a training job retries, it re-logs epochs the tracker already has, so key the tracker run to the job attempt and resume it, or discard the partial duplicate before reporting results. Retries are normal in training infrastructure. Trackers that treat every attempt as a new run, or that append without checking, turn one training run into a doubled history.