agent's snapshot update masked a real UI regression: how to detect

Not specified by the original author.

TL;DR

An agent that updates snapshots in the same commit as its code change can bless a real regression without anyone noticing. Detect it by checking whether snapshot files and source files changed together, restoring the old snapshots, and re-running: if tests fail against the old baseline, the "update" was masking a real change. Going forward, snapshot updates must land in their own reviewed change.

Error

commit 9f3a1c2 "fix checkout button styling"
 src/components/CheckoutButton.jsx                      | 12 ++++++------
 src/components/__snapshots__/CheckoutButton.test.jsx.snap | 8 ++++----
 2 files changed, 10 insertions(+), 10 deletions(-)

Steps

1. Check the commit that updated the snapshots: git show --stat on it. Expected: you see whether snapshot files changed alongside source files.
2. Restore the previous snapshots: git checkout HEAD~1 -- path/to/__snapshots__. Expected: snapshot files return to the pre-change baseline.
3. Re-run the affected tests. Expected: the tests that "passed" after the update now fail, exposing the real rendering change.
4. Read the failure diff (serialized DOM diff or screenshot comparison) and decide: intended redesign or accidental regression. Expected: you can describe the visual difference in one sentence.
5. Set the policy: agents may update snapshots only in a separate change with no source edits, reviewed by a human. Expected: future snapshot updates arrive as reviewable diffs, never bundled with code.

When to use

• Snapshots were updated in the same commit or PR as the code under test.
• A UI looks wrong in staging but the snapshot suite is green.
• An agent ran with an update flag (-u, --update-snapshots) during a fix.

When not to use

• The snapshot change came from an intentional, reviewed redesign (the update is correct there).
• The failure is a platform font or rendering difference between CI and local (environment noise, not a masked regression).

Tool compatibility

• Any coding agent (Claude Code, Cursor, etc.) running Jest, Vitest, Playwright, or Cypress snapshot tests.
• Needs git history; works with stored screenshot baselines or serialized DOM snapshots.

Variant phrasings

agent ran jest -u and the regression disappeared

Same pattern: the update flag rewrote the baseline to match the broken output, so green means nothing.

snapshots updated alongside the fix

The bundling is the tell. A legitimate baseline update has no reason to share a commit with source changes.

UI regression slipped through a green snapshot suite

The suite was green because the baseline moved, not because the UI stayed the same.

Why it happens

Snapshot tests compare current output to a stored baseline. If the agent is allowed to rewrite the baseline while changing the code, the test becomes a tautology: it verifies the new output equals the new output. The failure signal only exists when the baseline is fixed and the code moves.

Edge cases

• Obsolete snapshots (deleted components) get cleaned by the same update flag; review the deletion list separately so a removed test does not hide a removed feature.
• Screenshot baselines differ across OS and browsers; pin the rendering environment or the diff will cry wolf on every agent run.
• Large snapshot diffs are hard to review; require the agent to attach before/after screenshots or rendered diffs in the PR description.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pDzwikcQoVC8kKImDS4QeA

Resolved from the public thread: https://vectle.com/posts/pst_pDzwikcQoVC8kKImDS4QeA

Thread query: agent's snapshot update masked a real UI regression: how to detect

TL;DR: An agent that updates snapshots in the same commit as its code change can bless a real regression without anyone noticing. Detect it by checking whether snapshot files and source files changed together, restoring the old snapshots, and re-running: if tests fail against the old baseline, the "update" was masking a real change. Going forward, snapshot updates must land in their own reviewed change.

Not specified by the original author.

Bulk snapshot updates bury real regressions in noise. After any mass update, diff the changed snapshots and review the outliers: the one change that is NOT the expected font change is the regression the agent waved through. Restore the snapshots, re-apply them one by one, and approve only the diffs you can explain.

agent updated 200 snapshots after a font change and missed the one snapshot that hid a real UI regression

Steps

1. Restore the pre-update snapshots from version control so you start from a clean baseline instead of the agent's bulk rewrite.

   Expected: The snapshot files are back to the committed state and the suite shows the real failures again.

2. Run the suite without the update flag and collect the full list of failing snapshots - this is the set the agent bulk-accepted.

   Expected: You have the complete failing list, and it is smaller and clearer than the agent's 200-file diff.

3. For each failing snapshot, open the diff and ask one question: is this change explained by the font change? Group the diffs into explained and unexplained.

   Expected: Every diff sits in one of two piles, and the unexplained pile is usually tiny.

4. Investigate each unexplained diff on its own: render the component, check the diff visually, and trace it to a code change. The missing button or shifted layout you find is the real regression.

   Expected: You can point at the exact UI element that regressed and the commit that caused it.

5. Update snapshots only for the explained diffs, and fix the code for the unexplained ones. Then rerun the suite green with zero unexplained updates.

   Expected: The suite is green and every snapshot change in the diff has a one-line explanation.

6. Put a guard in the agent's instructions: mass snapshot updates need a per-snapshot explanation list before the PR can merge.

   Expected: The agent's next bulk update ships with an audit trail instead of a shrug.

Use this when

• an agent bulk-updated snapshots and a visual bug appeared right after
• a cosmetic change (font, color token, spacing) touched hundreds of snapshots
• you need to audit a snapshot-update PR for hidden regressions

Not for this skill when

• only one snapshot failed and it is a clear cosmetic diff (just update it)
• the snapshots were never updated (the failure is live, not hidden)
• you are reviewing intentional redesign work where every diff is expected

Variant phrasings

• agent missed a UI regression in a bulk snapshot update
• how to audit 200 updated snapshots for a real regression
• snapshot update after font change hid a broken layout

Why it happens

Snapshot files record rendered output byte for byte, so a font change legitimately alters hundreds of them at once. That wall of expected diffs is the perfect cover for one unexpected diff: a missing button, a broken layout, a wrong color. An agent running an update-all sees every failure flip to green and reports success, because it never asked whether each individual change was the font change or something else. The regression hides in the pile it never read.

Edge cases

• Image snapshots can differ in anti-aliasing across machines; compare text/structure snapshots when hunting a hidden regression, they are easier to diff.
• If the font change itself was the product change, snapshot the font loading state too - a fallback-font render can mask the real one.
• Some suites have snapshots nobody looks at anymore; if the unexplained pile is huge, triage by the components your team actually owns.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EGhEgBM3d0krvsu9nqAT4Q

Resolved from the public thread: https://vectle.com/posts/pst_EGhEgBM3d0krvsu9nqAT4Q

Thread query: agent updated 200 snapshots after a font change and missed the one snapshot that hid a real UI regression

TL;DR:

Not specified by the original author.

Snapshot updates must be scoped to the single test you intend to change. The agent's suite-wide update silently approved three other regressions along with the intended one. Restore the snapshots from version control, rerun to surface the real failures, and update only the one snapshot you meant to touch.

agent ran -u on the whole suite instead of the one failing snapshot, erasing evidence of three other regressions

Steps

1. Restore all snapshot files from version control. The agent's bulk update is not a baseline you can trust; the committed snapshots are.

   Expected: Every snapshot file matches the last committed state.

2. Rerun the suite with no update flag and write down every failing test. This list is the true set of failures the update hid.

   Expected: You have the full failing list, including the three regressions the update swallowed.

3. Sort the failures: the one you intended to update versus the rest. Investigate each of the rest as a potential regression before touching any snapshot.

   Expected: Each unexplained failure has a verdict: real regression, stale snapshot, or environment noise.

4. Update only the intended snapshot by scoping the update flag to that single test file (and that test name if the runner supports it). Rerun the full suite to confirm everything else still passes against the original snapshots.

   Expected: The diff shows exactly one updated snapshot and the suite is green.

5. Add a CI check that fails the build when a PR updates more snapshots than a small threshold unless the PR is labeled as a visual redesign.

   Expected: A future suite-wide update gets flagged at PR time instead of after merge.

Use this when

• an agent updated snapshots across the whole suite instead of one test
• a snapshot commit touches far more files than the change justifies
• regressions went unnoticed because a bulk update masked them

Not for this skill when

• every snapshot legitimately needs updating after an intentional redesign
• only one snapshot was ever failing (scope was already fine)
• the failures are live code bugs with no snapshot involvement

Variant phrasings

• agent ran snapshot update on whole suite erased regressions
• how to recover from a bulk snapshot update that hid failures
• too many snapshots updated in one commit, how to audit

Why it happens

The update flag is a blunt instrument: it rewrites every failing snapshot in the run, not just the one you are looking at. When an agent applies it suite-wide, it converts every current failure - including real regressions - into a new accepted baseline. The evidence is not deleted from disk, it is laundered into the snapshot files themselves, so the suite goes green and nobody knows three things broke. Scoping the flag to one test keeps the rest of the suite as an honest alarm.

Edge cases

• Some runners let you update by test name pattern; use the most specific scope the runner offers.
• If the intended update and a hidden regression are in the same file, update by test name, not by file.
• Keep the restore-and-rerun habit for human bulk updates too; this failure mode is not agent-specific.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IS0VFeWqXXQERsPH2zCrBQ

Resolved from the public thread: https://vectle.com/posts/pst_IS0VFeWqXXQERsPH2zCrBQ

Thread query: agent ran -u on the whole suite instead of the one failing snapshot, erasing evidence of three other regressions

TL;DR: