dvclive log failed error "dvclive: metrics file already exists and resume is false"

Not specified by the original author.

TL;DR

Decide whether to resume or to start clean, then set the flag to match: dvclive refuses to overwrite an existing metrics file when resume is false, so rerun with resume enabled to continue the run, or remove the stale dvclive directory to start a genuinely new run. The metrics file belongs to a previous run in the same workspace. dvclive protects it because silently overwriting would mix two runs in one history.

dvclive log failed error "dvclive: metrics file already exists and resume is false"

Use this when

• The check or run fails with this symptom: a dvclive log fails with metrics file already exists and resume is false.
• You are working on a dvclive instrumented training script rerun in the same workspace and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A pipeline managed by DVC experiments, where the workspace is reset per experiment and a leftover file signals a failed cleanup.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Check the dvclive directory timestamps and the run that created it. Verify: the metrics file belongs to an identified earlier run
2. If continuing that run, initialise dvclive with resume enabled. Verify: logging appends to the existing history
3. If starting fresh, remove or archive the old dvclive output first. Verify: a new metrics file is created cleanly
4. Rerun the training script. Verify: metrics log without the error
5. In automated pipelines, create the dvclive directory per run or clean it in the run setup. Verify: repeat runs stop colliding

Variant phrasings

dvclive metrics file already exists resume is false

The shorter phrasing focused on the named failure.

dvclive log failed metrics exists error

The symptom phrasing a searcher tries next.

dvclive resume metrics file fix

The variant naming the failing stage or tool directly.

Compatibility: dvclive logging and resume behaviour, DVC experiment workspaces

Why it happens

dvclive writes metrics to a fixed location relative to the run. Workspaces reused across runs, the normal case outside DVC experiment sandboxes, keep the previous metrics file, and the library chooses an explicit error over corrupting the earlier run history.

Edge cases / pitfalls

• Resuming a run whose parameters changed produces a single history with two configurations; start fresh when the configuration changed.
• Archiving the old directory beats deleting it when the earlier run was never pushed anywhere.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4ovv8lsqFrNy1BRNi360-Q

Resolved from the public thread: https://vectle.com/posts/pst_4ovv8lsqFrNy1BRNi360-Q

Thread query: dvclive log failed error "dvclive: metrics file already exists and resume is false"

TL;DR: Decide whether to resume or to start clean, then set the flag to match: dvclive refuses to overwrite an existing metrics file when resume is false, so rerun with resume enabled to continue the run, or remove the stale dvclive directory to start a genuinely new run. The metrics file belongs to a previous run in the same workspace. dvclive protects it because silently overwriting would mix two runs in one history.

Not specified by the original author.

TL;DR

Recover metrics from the run, not from the truncated file alone: dvclive metrics JSON was cut mid-write by the training crash, so salvage the complete records before the truncation point, and treat the run as crashed rather than reading the file as finished. A truncated JSON file holds a true prefix and a missing rest. Reporting from it without flagging the crash understates the run and hides the failure that matters.

agent couldnt parse dvclive metrics json after training crash truncated file error

Use this when

• The check or run fails with this symptom: an agent cannot parse dvclive metrics JSON after a training crash truncated the file.
• You are working on an agent workflow reading dvclive metrics from completed and crashed runs and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A complete file that fails parsing for a different reason, such as a hand edit, which needs the edit reverted rather than crash handling.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Check the training run status and logs for the crash. Verify: the run ended abnormally at the truncation time
2. Parse the file incrementally and keep the records that decode fully. Verify: the salvageable metric history is recovered with its last complete step
3. Mark the run as crashed in the workflow output, with the crash point. Verify: downstream summaries exclude it from completed comparisons
4. Fix the logging to write metrics through dvclive normal flush behaviour and to close the file on exit handlers where possible. Verify: a later crash leaves a cleaner prefix
5. Rerun or resume the training as the project decides. Verify: a complete metrics file replaces the truncated one

Variant phrasings

agent couldnt parse dvclive metrics json training crash truncated

The shorter phrasing focused on the named failure.

dvclive metrics truncated file parse error

The symptom phrasing a searcher tries next.

truncated dvclive metrics salvage fix

The variant naming the failing stage or tool directly.

Compatibility: dvclive metrics files, crash recovery in agent run analysis

Why it happens

dvclive appends metrics as training proceeds, and a hard crash, a kill, or a spot reclamation can stop the process mid-write. The resulting file is not corrupt data; it is an interrupted stream, and parsers that demand a whole valid document report a format error instead of a crashed run.

Edge cases / pitfalls

• Do not pad or close the JSON by hand and present it as the run record; label salvaged data as partial wherever it is used.
• The last complete step is a useful lower bound for progress reporting, never a final result.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qZPRNNi4k75oHVwSRMllAQ

Resolved from the public thread: https://vectle.com/posts/pst_qZPRNNi4k75oHVwSRMllAQ

Thread query: agent couldnt parse dvclive metrics json after training crash truncated file error

TL;DR: Recover metrics from the run, not from the truncated file alone: dvclive metrics JSON was cut mid-write by the training crash, so salvage the complete records before the truncation point, and treat the run as crashed rather than reading the file as finished. A truncated JSON file holds a true prefix and a missing rest. Reporting from it without flagging the crash understates the run and hides the failure that matters.

Not specified by the original author.

TL;DR

Make the retry resume the same run instead of opening a new logging session: when a training job retries, it re-logs epochs the tracker already has, so key the tracker run to the job attempt and resume it, or discard the partial duplicate before reporting results. Retries are normal in training infrastructure. Trackers that treat every attempt as a new run, or that append without checking, turn one training run into a doubled history.

experiment metrics logged twice after training job retry duplicates run history error

Use this when

• The check or run fails with this symptom: experiment metrics are logged twice after a training job retry and duplicate the run history.
• You are working on a tracked training job running on retrying infrastructure such as a scheduler or a spot instance and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• Two genuinely separate runs with identical parameters, which should both exist and be distinguished by their run metadata.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Compare the duplicated run records: timestamps, attempt identifiers, and step ranges. Verify: the duplicates share one job and overlapping steps
2. Check how the job creates its tracker run on start. Verify: each attempt creates or resumes without checking for an existing run
3. Pass a stable run identifier from the job, and resume that run on retry, or end the failed attempt run as failed and start reporting from the successful one only. Verify: a retry produces one coherent history
4. Clean the existing duplicate by removing the partial attempt record. Verify: reports show a single run
5. Add attempt number and job id as run tags. Verify: future duplicates are recognisable at a glance

Variant phrasings

experiment metrics logged twice training job retry duplicates

The shorter phrasing focused on the named failure.

duplicate run history after retry error

The symptom phrasing a searcher tries next.

training retry duplicate metrics fix

The variant naming the failing stage or tool directly.

Compatibility: MLflow, W&B, and similar trackers with run resume, retrying training schedulers

Why it happens

Training frameworks log per epoch with no knowledge of job attempts, and trackers accept the writes at face value. A retry after a preemption replays the early epochs into the same or a sibling run, and every downstream average, chart, and best-run selection inherits the duplication.

Edge cases / pitfalls

• Deduplicating by step value is unsafe; identical losses can occur legitimately. Dedupe by attempt identity.
• A resumed run must not re-log parameters; most trackers reject or duplicate them, compounding the original problem.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstj3iTCI71kM--Fqggesm9w

Resolved from the public thread: https://vectle.com/posts/pstj3iTCI71kM--Fqggesm9w

Thread query: experiment metrics logged twice after training job retry duplicates run history error

TL;DR: Make the retry resume the same run instead of opening a new logging session: when a training job retries, it re-logs epochs the tracker already has, so key the tracker run to the job attempt and resume it, or discard the partial duplicate before reporting results. Retries are normal in training infrastructure. Trackers that treat every attempt as a new run, or that append without checking, turn one training run into a doubled history.