agent's diff of two executed notebooks stripped only output noise: no real changes

TL;DR

Two executed notebooks differ in hundreds of places that do not matter: execution counts increment, timestamps shift, and widget state churns, so a raw JSON diff flags every cell as changed. Diff with a notebook-aware tool that ignores volatile metadata and compares code cells structurally. Configure the ignore rules once (execution counts, outputs, metadata noise) and the diff shows only the changes a reviewer cares about.

agent's diff of two executed notebooks stripped only output noise: no real changes

Use this when

• A diff of two notebook versions shows every cell changed when only one line of code moved.
• An agent's change-review reports massive diffs for trivial notebook edits.
• Code review on notebooks is impractical because of output and metadata churn.

Not for this skill when

• The notebooks genuinely differ in logic across many cells. That is a real large diff, not noise.
• Git merge conflicts inside notebook JSON. That is a merge problem, a different fix.

Steps

1. Confirm the noise: diff the two notebooks as raw JSON and note the change classes (execution_count, output timestamps, widget state). Expected: you can list which volatile fields dominate the diff.
2. Install a notebook-aware diff tool (nbdime) and run it on the pair. Expected: the structural diff is dramatically smaller than the raw JSON diff.
3. Configure the ignore rules for your workflow: ignore execution counts, ignore outputs when reviewing code changes, keep outputs when reviewing results. Expected: a config file or alias encoding the team's diff policy.
4. Re-run the diff with the policy applied. Expected: only real source and meaningful metadata changes appear.
5. Wire it into the agent's review step: the agent diffs with the notebook-aware tool and summarizes the structural changes. Expected: change reports describe code changes, not output churn.
6. For version control, combine with output stripping on commit so future diffs start clean. Expected: committed notebooks diff on code by default.

Tool compatibility

nbdime 4.x; git 2.x with nbdime diffdriver; notebook 6.x/7.x and JupyterLab 3.x/4.x.

Variant phrasings

diff two jupyter notebooks ignores outputs

The how-to phrasing. Use a notebook-aware diff with output-ignoring rules.

notebook diff shows every cell changed

The symptom phrasing. Volatile metadata is the noise; strip it from the diff.

nbdime diff execution count noise

The tool phrasing. Configure nbdime to ignore the volatile fields.

Why it happens

Notebook JSON mixes three things with different stability: source code (stable), execution metadata like counts and timestamps (volatile by design), and outputs (regenerated on every run). A text diff cannot tell them apart, so it reports all three as changes, and reviewers learn to ignore notebook diffs entirely, which is how real bugs slip through. Notebook-aware diff tools parse the structure and let you declare which parts are semantically meaningful. The agent's mistake in the query, stripping only output noise, left the metadata noise in place, which is why 'no real changes' still looked like everything changed.

Edge cases / pitfalls

• Ignoring outputs while reviewing a results change hides the actual result. Match the diff policy to the review question: code diff for logic, output diff for results.
• Execution counts can be meaningful in one case: proving top-to-bottom execution order. Do not ignore them when order is the question.
• Merge drivers for notebooks exist but are not magic. Complex concurrent edits to one notebook still need human judgment.
• Agents that auto-accept 'empty' diffs should double-check with outputs included before declaring two notebooks equivalent.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstMargel0r9YpfTslcWxq-w