agent couldnt parse dvclive metrics json after training crash truncated file error
Recover metrics from the run, not from the truncated file alone: dvclive metrics JSON was cut mid-write by the training crash, so salvage the complete records before the truncation point, and treat the run as crashed rather than reading the file as finished. Use when an agent workflow reading dvclive metrics from completed and crashed runs. Not for a complete file that fails parsing for a different reason, such as
TL;DR
Recover metrics from the run, not from the truncated file alone: dvclive metrics JSON was cut mid-write by the training crash, so salvage the complete records before the truncation point, and treat the run as crashed rather than reading the file as finished. A truncated JSON file holds a true prefix and a missing rest. Reporting from it without flagging the crash understates the run and hides the failure that matters.
agent couldnt parse dvclive metrics json after training crash truncated file errorUse this when
- The check or run fails with this symptom: an agent cannot parse dvclive metrics JSON after a training crash truncated the file.
- You are working on an agent workflow reading dvclive metrics from completed and crashed runs and the failure repeats on the same data or configuration.
- You need the failing run, call, or stage isolated before changing code or thresholds.
Not for this skill when
- A complete file that fails parsing for a different reason, such as a hand edit, which needs the edit reverted rather than crash handling.
- The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.
Steps
- Check the training run status and logs for the crash. Verify: the run ended abnormally at the truncation time
- Parse the file incrementally and keep the records that decode fully. Verify: the salvageable metric history is recovered with its last complete step
- Mark the run as crashed in the workflow output, with the crash point. Verify: downstream summaries exclude it from completed comparisons
- Fix the logging to write metrics through dvclive normal flush behaviour and to close the file on exit handlers where possible. Verify: a later crash leaves a cleaner prefix
- Rerun or resume the training as the project decides. Verify: a complete metrics file replaces the truncated one
Variant phrasings
agent couldnt parse dvclive metrics json training crash truncated
The shorter phrasing focused on the named failure.
dvclive metrics truncated file parse error
The symptom phrasing a searcher tries next.
truncated dvclive metrics salvage fix
The variant naming the failing stage or tool directly.
Compatibility: dvclive metrics files, crash recovery in agent run analysis
Why it happens
dvclive appends metrics as training proceeds, and a hard crash, a kill, or a spot reclamation can stop the process mid-write. The resulting file is not corrupt data; it is an interrupted stream, and parsers that demand a whole valid document report a format error instead of a crashed run.
Edge cases / pitfalls
- Do not pad or close the JSON by hand and present it as the run record; label salvaged data as partial wherever it is used.
- The last complete step is a useful lower bound for progress reporting, never a final result.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_qZPRNNi4k75oHVwSRMllAQ