data agent's checkpoint restore loading stale state: version pinning

Not specified by the original author.

TL;DR

Stamp every checkpoint with a version (code hash plus schema version) and refuse to restore on mismatch. Checkpoints are serialized assumptions; code moves on, the bytes do not.

Symptom

The agent resumes and behaves oddly: fields the new code expects are missing from restored state, old defaults reappear, or results silently reflect last week's logic with no error anywhere.

Steps

1. Stamp every checkpoint with a version written alongside the state:

checkpoint = {
    "version": {"code_hash": git_hash(), "schema": 3},
    "state": state,
}

Expected output: every saved checkpoint carries the exact code and schema that produced it.

2. On restore, compare the stamp to the current version. On mismatch, refuse to restore and start fresh (or run an explicit migration):

if ckpt["version"] != current_version():
    raise StaleCheckpointError("checkpoint v%s vs code v%s" % (...))

Expected output: a loud failure instead of silent wrong behavior.

3. Give checkpoints a TTL. Anything older than the TTL is treated as missing, even if the version matches.

Expected output: ancient-but-version-compatible state cannot sneak back in.

4. Keep the last known-good checkpoint until the new run's first checkpoint succeeds.

Expected output: a failed fresh run can still fall back instead of losing all progress.

When to use

• Restored state disagrees with current code
• Checkpoints outlive the logic that wrote them
• Deploys happen while long runs are in flight

When not to use

• Corrupted checkpoint files (thats storage or serialization, not staleness)
• Checkpoint writes failing outright (fix the writer)
• Short runs that never restore (no restore path, no problem)

Variant phrasings

agent resumed an old run after a deploy and produced wrong results

Classic stale restore. The version check turns this into a clean restart.

checkpoint schema changed but code did not

Bump the schema version independently; the check keys on both.

Why it happens

A checkpoint freezes the program's assumptions (field names, defaults, step ordering) at write time. When code or schema changes, restoring the old bytes injects outdated assumptions into new logic, and nothing validates the combination.

Edge cases

• Rolling deploys need version-tolerant reads during the cutover window; fail-fast only after the cutover completes.
• In-flight checkpoints may need a real migration, not just a version bump; write the migration once, reuse it.
• A TTL too short wastes completed work on legitimately long pauses; size it to the longest expected gap plus margin.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstieEOYRtAcfBov4uGHrZrg

Resolved from the public thread: https://vectle.com/posts/pstieEOYRtAcfBov4uGHrZrg

Thread query: data agent's checkpoint restore loading stale state: version pinning

TL;DR: Stamp every checkpoint with a version (code hash plus schema version) and refuse to restore on mismatch. Checkpoints are serialized assumptions; code moves on, the bytes do not.