agent's notebook JSON diff failed: flagged every cell as changed, wrong diff

TL;DR

A diff that flags every cell as changed is diffing the wrong things: execution counts, timestamps, and output metadata change on every run, so a naive JSON diff reports the whole notebook as modified. Diff structurally: match cells by id, compare source (and outputs if you care about them), and ignore volatile fields like execution_count. The diff should show what the human changed, not what the kernel touched.

agent's notebook JSON diff failed: flagged every cell as changed, wrong diff

Use this when

• An agent's notebook diff shows all cells changed after a trivial edit.
• Diffs are unreadable due to execution-count noise.
• The agent diffs raw JSON.

Not for this skill when

• The diff misses real changes. That is a matching bug.
• A human diffs in git. Use nbdime.

Steps

1. Match cells by id across the two versions, not by position. Verify: moved cells are detected as moved, not changed.
2. Compare source for content changes; compare outputs only if the diff cares about them. Verify: a re-executed notebook shows no content diff.
3. Ignore volatile fields: execution_count, execution timestamps, output metadata. Verify: the ignore list is explicit.
4. Render the diff as cell-by-cell source diffs. Verify: a one-line edit shows as a one-line diff.
5. Consider nbdime for the heavy lifting. Verify: its output matches your custom diff on test cases.

Variant phrasings

notebook diff every cell changed

Short form. Ignore volatile metadata.

nbdime vs json diff notebook

The tool phrasing. nbdime exists for this.

agent diff ipynb noisy

The agent phrasing. Match by id, compare source.

Compatibility: nbformat 5.x cell ids, nbdime. The volatile-field list is version-stable.

Why it happens

Notebook JSON mixes content (source) with run artifacts (counts, outputs, timestamps). A text diff cannot tell them apart, so re-executing a notebook 'changes' every cell. Humans learn to ignore this; agents must be taught explicitly, by diffing the content model instead of the JSON text. Cell ids (v4.5+) make matching robust against reordering.

Edge cases / pitfalls

• Position-based matching breaks when cells are reordered. Always match by id with a position fallback.
• Outputs are content for some diffs (did the figure change?) and noise for others. Make it a flag.
• Merges need the same discipline: volatile fields cause spurious conflicts. Configure the merge driver accordingly.
• A diff that hides too much is as bad as a noisy one. Show what you ignore in a summary line.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_dJo02yFqSWXwG4kKBFO6QA