agent's snapshot diff crashed with json parse error
Fixes snapshot-diff crashes on malformed JSON: parse defensively, stream large snapshots, and quarantine bad segments. Use when diff jobs die parsing. Not for semantic diff issues.
TL;DR
The snapshot contains malformed JSON (truncated writes, bad escaping, mixed encodings) and the diff's strict parser dies on it. Parse with error recovery, stream instead of loading everything, and quarantine the bad segments for inspection. A diff tool must survive ugly input.
Error
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)Steps
- Find the bad segment: binary-search the snapshot file for the offset where parsing fails. Expected: the exact corrupt region.
- Decide: repair it (if the corruption pattern is known) or quarantine it and diff the rest. Expected: an explicit call, not a crash.
- Switch the diff to streaming parse (one record at a time) instead of loading the whole snapshot. Expected: memory stays flat and one bad record cannot kill the run.
- Wrap per-record parsing in error handling that logs and continues. Expected: the diff completes with a quarantine list.
- Fix the snapshot writer so it cannot produce truncated JSON (atomic writes, checksums). Expected: the corruption stops at the source.
When to use
- Diff jobs crash with JSON parse errors.
- An agent snapshots CRM state and diffs it later.
- Large snapshots that may have been truncated.
When not to use
- Diffs that complete but look wrong (semantic issue).
- Schema changes between snapshots (different problem).
Tool compatibility
- Streaming JSON parsers; JSON lines format.
- Snapshot writers in the agent.
Variant phrasings
JSONDecodeError on snapshot
Find the bad segment; parse defensively.
diff crashed partway
Stream and quarantine.
Why it happens
Snapshots are written by one process and read much later by another. Anything that interrupts the write (crash, timeout, disk full) leaves truncated JSON that a strict parser rejects wholesale.
Edge cases
- JSON lines (one object per line) isolates corruption to single lines; prefer it for snapshots.
- Atomic writes (write temp, then rename) prevent torn files.
- Checksums on snapshot files catch corruption before the diff starts.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstcxdWDYU2bovigJFzgpbwA