TensorBoard event file corrupted error 'DataLossError: truncated record at' failed to load

Not specified by the original author.

TL;DR

A truncated record means the event file ends mid-record, almost always because the writing process died or the file was copied while being written: the data up to the truncation is readable, so copy the file, read it with a tolerant reader that stops at the bad record, and treat the run's tail as lost. Prevent recurrence by copying event files only after the writer closes them.

TensorBoard event file corrupted error 'DataLossError: truncated record at' failed to load

Use this when

• The run or call fails with this symptom: TensorBoard event file corrupted: DataLossError: truncated record at, when loading.
• You are working on a TensorBoard run whose event file fails to load with a truncation error and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A file truncated by a full disk rather than a dead writer, where the fix also includes the disk problem.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Copy the event file to a working location so the original stays untouched for any recovery attempt. Verify: a working copy exists
2. Read the copy with a tolerant event iterator and note the last step and timestamp successfully read. Verify: the readable prefix's extent is known
3. Confirm the cause from the job's history: killed process, disk full, or a live copy during writing. Verify: the truncation cause is identified
4. Load the recovered prefix in TensorBoard or export its scalars, and mark the run as partial in any report. Verify: the surviving data is usable and labelled partial

Variant phrasings

tensorboard dataloss truncated record event file

The verbatim phrasing.

event file corrupted truncated record recover prefix

The recovery phrasing.

tensorboard run fails to load truncated events fix

The fix phrasing.

Compatibility: TensorBoard event file reading and recovery

Why it happens

Event files are append-only streams of length-prefixed records, so an interrupted write damages exactly the tail. Readers that demand a clean end-of-file refuse the whole file, but the stream format itself lets a tolerant reader recover everything before the torn record.

Edge cases / pitfalls

• Live dashboards reading a file still being written can show this transiently; it clears when the writer flushes.
• Sync tools that copy open files manufacture truncations; snapshot closed files only.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vOQKIm48ciTQv-B-Y3ug5A

Resolved from the public thread: https://vectle.com/posts/pst_vOQKIm48ciTQv-B-Y3ug5A

Thread query: TensorBoard event file corrupted error 'DataLossError: truncated record at' failed to load

TL;DR: A truncated record means the event file ends mid-record, almost always because the writing process died or the file was copied while being written: the data up to the truncation is readable, so copy the file, read it with a tolerant reader that stops at the bad record, and treat the run's tail as lost. Prevent recurrence by copying event files only after the writer closes them.