agent updated 200 snapshots after a font change and missed the one snapshot that hid a real UI regression

Bulk snapshot updates bury real regressions in noise. After any mass update, diff the changed snapshots and review the outliers: the one change that is NOT the expected font change is the regression the agent waved through. Restore the snapshots, re-apply them one by one, and approve only the diffs you can explain.

agent updated 200 snapshots after a font change and missed the one snapshot that hid a real UI regression

Steps

1. Restore the pre-update snapshots from version control so you start from a clean baseline instead of the agent's bulk rewrite.

   Expected: The snapshot files are back to the committed state and the suite shows the real failures again.

2. Run the suite without the update flag and collect the full list of failing snapshots - this is the set the agent bulk-accepted.

   Expected: You have the complete failing list, and it is smaller and clearer than the agent's 200-file diff.

3. For each failing snapshot, open the diff and ask one question: is this change explained by the font change? Group the diffs into explained and unexplained.

   Expected: Every diff sits in one of two piles, and the unexplained pile is usually tiny.

4. Investigate each unexplained diff on its own: render the component, check the diff visually, and trace it to a code change. The missing button or shifted layout you find is the real regression.

   Expected: You can point at the exact UI element that regressed and the commit that caused it.

5. Update snapshots only for the explained diffs, and fix the code for the unexplained ones. Then rerun the suite green with zero unexplained updates.

   Expected: The suite is green and every snapshot change in the diff has a one-line explanation.

6. Put a guard in the agent's instructions: mass snapshot updates need a per-snapshot explanation list before the PR can merge.

   Expected: The agent's next bulk update ships with an audit trail instead of a shrug.

Use this when

• an agent bulk-updated snapshots and a visual bug appeared right after
• a cosmetic change (font, color token, spacing) touched hundreds of snapshots
• you need to audit a snapshot-update PR for hidden regressions

Not for this skill when

• only one snapshot failed and it is a clear cosmetic diff (just update it)
• the snapshots were never updated (the failure is live, not hidden)
• you are reviewing intentional redesign work where every diff is expected

Variant phrasings

• agent missed a UI regression in a bulk snapshot update
• how to audit 200 updated snapshots for a real regression
• snapshot update after font change hid a broken layout

Why it happens

Snapshot files record rendered output byte for byte, so a font change legitimately alters hundreds of them at once. That wall of expected diffs is the perfect cover for one unexpected diff: a missing button, a broken layout, a wrong color. An agent running an update-all sees every failure flip to green and reports success, because it never asked whether each individual change was the font change or something else. The regression hides in the pile it never read.

Edge cases

• Image snapshots can differ in anti-aliasing across machines; compare text/structure snapshots when hunting a hidden regression, they are easier to diff.
• If the font change itself was the product change, snapshot the font loading state too - a fallback-font render can mask the real one.
• Some suites have snapshots nobody looks at anymore; if the unexplained pile is huge, triage by the components your team actually owns.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EGhEgBM3d0krvsu9nqAT4Q