agent ran -u on the whole suite instead of the one failing snapshot, erasing evidence of three other regressions

Snapshot updates must be scoped to the single test you intend to change. The agent's suite-wide update silently approved three other regressions along with the intended one. Restore the snapshots from version control, rerun to surface the real failures, and update only the one snapshot you meant to touch.

agent ran -u on the whole suite instead of the one failing snapshot, erasing evidence of three other regressions

Steps

1. Restore all snapshot files from version control. The agent's bulk update is not a baseline you can trust; the committed snapshots are.

Expected: Every snapshot file matches the last committed state.

1. Rerun the suite with no update flag and write down every failing test. This list is the true set of failures the update hid.

Expected: You have the full failing list, including the three regressions the update swallowed.

1. Sort the failures: the one you intended to update versus the rest. Investigate each of the rest as a potential regression before touching any snapshot.

Expected: Each unexplained failure has a verdict: real regression, stale snapshot, or environment noise.

1. Update only the intended snapshot by scoping the update flag to that single test file (and that test name if the runner supports it). Rerun the full suite to confirm everything else still passes against the original snapshots.

Expected: The diff shows exactly one updated snapshot and the suite is green.

1. Add a CI check that fails the build when a PR updates more snapshots than a small threshold unless the PR is labeled as a visual redesign.

Expected: A future suite-wide update gets flagged at PR time instead of after merge.

Use this when

• an agent updated snapshots across the whole suite instead of one test
• a snapshot commit touches far more files than the change justifies
• regressions went unnoticed because a bulk update masked them

Not for this skill when

• every snapshot legitimately needs updating after an intentional redesign
• only one snapshot was ever failing (scope was already fine)
• the failures are live code bugs with no snapshot involvement

Variant phrasings

• agent ran snapshot update on whole suite erased regressions
• how to recover from a bulk snapshot update that hid failures
• too many snapshots updated in one commit, how to audit

Why it happens

The update flag is a blunt instrument: it rewrites every failing snapshot in the run, not just the one you are looking at. When an agent applies it suite-wide, it converts every current failure - including real regressions - into a new accepted baseline. The evidence is not deleted from disk, it is laundered into the snapshot files themselves, so the suite goes green and nobody knows three things broke. Scoping the flag to one test keeps the rest of the suite as an honest alarm.

Edge cases

• Some runners let you update by test name pattern; use the most specific scope the runner offers.
• If the intended update and a hidden regression are in the same file, update by test name, not by file.
• Keep the restore-and-rerun habit for human bulk updates too; this failure mode is not agent-specific.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IS0VFeWqXXQERsPH2zCrBQ