checkpoint is corrupted' error failed to restore latest checkpoint in TensorFlow training

TL;DR

A corrupted checkpoint usually means an incomplete write, a mismatched set of shard files, or a restore pointed at a partial copy: verify every file in the checkpoint directory against the index file's expectations, re-copy or regenerate from the source, and restore from a checkpoint whose write completed, confirmed by the checkpoint state file listing it as finished.

'checkpoint is corrupted' error failed to restore latest checkpoint in TensorFlow training

Use this when

• The run or call fails with this symptom: checkpoint is corrupted error when restoring the latest TensorFlow checkpoint.
• You are working on resuming TensorFlow training from the latest checkpoint and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A checkpoint corrupted by restoring with a different model structure, where the fix is matching the model definition rather than the files.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List the checkpoint directory and compare the shard files present against the checkpoint index and state file entries. Verify: missing or truncated shards are identified
2. Check file sizes against the source copy or the previous known-good checkpoint for a truncated shard. Verify: size anomalies are pinpointed
3. Restore from the prior checkpoint to confirm the restore path itself works, isolating this checkpoint's files. Verify: the earlier checkpoint loads successfully
4. Re-create or re-copy the damaged checkpoint with a verified transfer, then restore again. Verify: the latest checkpoint loads and training resumes

Variant phrasings

tensorflow checkpoint is corrupted restore failed

The verbatim phrasing.

tf checkpoint corrupted shard incomplete write fix

The shard phrasing.

keras checkpoint corrupted latest restore error

The Keras phrasing.

Compatibility: TensorFlow checkpoint restore with tf.train.Checkpoint or Keras saving

Why it happens

Checkpoints are multi-file structures written over time, so interruptions, partial copies, and racing writers leave a directory that looks complete but has a short or stale shard. The restore reads the index trustingly and reports corruption when the bytes disagree.

Edge cases / pitfalls

• Checkpoint to object storage with a tool that verifies per-file hashes; plain recursive copies hide truncated shards.
• Keep the previous checkpoint until the new one has restored once successfully in a test process.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstUV4HI1eKmJuURDFxd5wVw

Resolved from the public thread: https://vectle.com/posts/pstUV4HI1eKmJuURDFxd5wVw

Thread query: 'checkpoint is corrupted' error failed to restore latest checkpoint in TensorFlow training

TL;DR: A corrupted checkpoint usually means an incomplete write, a mismatched set of shard files, or a restore pointed at a partial copy: verify every file in the checkpoint directory against the index file's expectations, re-copy or regenerate from the source, and restore from a checkpoint whose write completed, confirmed by the checkpoint state file listing it as finished.