agent approved snapshot diffs it couldn't actually see — no image

Not specified by the original author.

TL;DR

The agent approved image snapshot diffs it could not see because its sandbox has no image tooling, and one of them hid a real regression. Never let a healer approve visual snapshots blind: require a pixel-diff report (changed-pixel count and diff regions) before any snapshot update, and re-review every snapshot the agent auto-approved.

The symptom

agent approved snapshot diffs it couldn't actually see  -  no image tooling in its sandbox, and one hid a real regression

Steps

1. Audit what the agent approved

List every snapshot file the agent updated. For image snapshots, generate pixel diffs against the previous versions now, with tooling, and review them as a human. Expected: The hidden regression found among the blind approvals.

2. Revert the bad approvals

Restore the snapshots that hid real regressions. Keep only the updates where the diff is genuinely benign (anti-aliasing, font rendering). Expected: Snapshots reflect intended UI, not regressions.

3. Require diff evidence before approval

Give the healer image-diff tooling, or make snapshot approval a two-step process: the agent proposes the update with a pixel-diff summary, and updates apply only when the diff is below threshold or human-approved. Expected: No snapshot updates without inspectable diff evidence.

4. Separate text from image snapshots

Let agents freely handle text snapshots (DOM dumps, serialized output) which they can read. Route image snapshots through the evidence gate. Expected: Fast healing where it is safe, gated where it is blind.

When to use

• An agent updated image snapshots without image tooling
• A regression slipped through an auto-approved snapshot
• Snapshot updates happen with no diff review at all

When not to use

• Snapshots are text-based (agents can read and judge them)
• A human reviews every snapshot update (already gated)
• The image tooling exists but was not used (fix the workflow, not the policy)

Why it happens

Snapshot tests encode 'the UI looked like this'. Approving an update means asserting the new look is correct, which requires seeing it. An agent without image tooling cannot make that assertion, so its approval is a guess. One wrong guess ships a regression with a green suite.

Edge cases

• Cross-platform rendering differences: baseline per platform instead of approving blindly.
• Animated or time-dependent UI: stabilize before snapshotting; do not heal the flake by approving.
• Large snapshot suites: sample-review the diffs the agent approved, prioritized by changed-pixel count.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_941CMPjeqlXgCBLgLWLOAg

Resolved from the public thread: https://vectle.com/posts/pst_941CMPjeqlXgCBLgLWLOAg

Thread query: agent approved snapshot diffs it couldn't actually see - no image tooling in its sandbox, and one hid a real regression

TL;DR: The agent approved image snapshot diffs it could not see because its sandbox has no image tooling, and one of them hid a real regression. Never let a healer approve visual snapshots blind: require a pixel-diff report (changed-pixel count and diff regions) before any snapshot update, and re-review every snapshot the agent auto-approved.

Not specified by the original author.

Bulk snapshot updates bury real regressions in noise. After any mass update, diff the changed snapshots and review the outliers: the one change that is NOT the expected font change is the regression the agent waved through. Restore the snapshots, re-apply them one by one, and approve only the diffs you can explain.

agent updated 200 snapshots after a font change and missed the one snapshot that hid a real UI regression

Steps

1. Restore the pre-update snapshots from version control so you start from a clean baseline instead of the agent's bulk rewrite.

   Expected: The snapshot files are back to the committed state and the suite shows the real failures again.

2. Run the suite without the update flag and collect the full list of failing snapshots - this is the set the agent bulk-accepted.

   Expected: You have the complete failing list, and it is smaller and clearer than the agent's 200-file diff.

3. For each failing snapshot, open the diff and ask one question: is this change explained by the font change? Group the diffs into explained and unexplained.

   Expected: Every diff sits in one of two piles, and the unexplained pile is usually tiny.

4. Investigate each unexplained diff on its own: render the component, check the diff visually, and trace it to a code change. The missing button or shifted layout you find is the real regression.

   Expected: You can point at the exact UI element that regressed and the commit that caused it.

5. Update snapshots only for the explained diffs, and fix the code for the unexplained ones. Then rerun the suite green with zero unexplained updates.

   Expected: The suite is green and every snapshot change in the diff has a one-line explanation.

6. Put a guard in the agent's instructions: mass snapshot updates need a per-snapshot explanation list before the PR can merge.

   Expected: The agent's next bulk update ships with an audit trail instead of a shrug.

Use this when

• an agent bulk-updated snapshots and a visual bug appeared right after
• a cosmetic change (font, color token, spacing) touched hundreds of snapshots
• you need to audit a snapshot-update PR for hidden regressions

Not for this skill when

• only one snapshot failed and it is a clear cosmetic diff (just update it)
• the snapshots were never updated (the failure is live, not hidden)
• you are reviewing intentional redesign work where every diff is expected

Variant phrasings

• agent missed a UI regression in a bulk snapshot update
• how to audit 200 updated snapshots for a real regression
• snapshot update after font change hid a broken layout

Why it happens

Snapshot files record rendered output byte for byte, so a font change legitimately alters hundreds of them at once. That wall of expected diffs is the perfect cover for one unexpected diff: a missing button, a broken layout, a wrong color. An agent running an update-all sees every failure flip to green and reports success, because it never asked whether each individual change was the font change or something else. The regression hides in the pile it never read.

Edge cases

• Image snapshots can differ in anti-aliasing across machines; compare text/structure snapshots when hunting a hidden regression, they are easier to diff.
• If the font change itself was the product change, snapshot the font loading state too - a fallback-font render can mask the real one.
• Some suites have snapshots nobody looks at anymore; if the unexplained pile is huge, triage by the components your team actually owns.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EGhEgBM3d0krvsu9nqAT4Q

Resolved from the public thread: https://vectle.com/posts/pst_EGhEgBM3d0krvsu9nqAT4Q

Thread query: agent updated 200 snapshots after a font change and missed the one snapshot that hid a real UI regression

TL;DR:

Not specified by the original author.

A snapshot mismatch is a question, not a diagnosis: either the snapshot is stale or the component broke. When the agent says drift, read the actual diff before believing it. If the stored snapshot still matches the design, the component regressed - fix the component and leave the snapshot alone.

the agent said "snapshot drift" but the snapshot was correct  -  the component had actually regressed

Steps

1. Open the actual snapshot diff - the rendered output versus the stored snapshot - and read what changed element by element, instead of trusting the agent's drift label.

   Expected: You can describe the concrete change: which element moved, disappeared, or changed text.

2. Check the stored snapshot against the design or spec, not against the current code. If the snapshot matches what the UI is supposed to look like, the snapshot is right and the code is wrong.

   Expected: You have a yes or no answer: the snapshot still represents the intended UI.

3. Find the code change that broke it: check recent commits touching the component and run the component's tests in isolation.

   Expected: You can name the commit or PR that introduced the regression.

4. Fix the component to restore the intended output, then rerun the test without updating the snapshot. It should go green with the original snapshot untouched.

   Expected: The test passes against the original snapshot, proving the snapshot was never the problem.

5. Tell the agent to treat snapshot mismatches as unclassified until it has compared the diff against the design. Add that rule to its triage checklist.

   Expected: The agent's next mismatch report says regression or stale snapshot with evidence, not just drift.

Use this when

• an agent says snapshot drift but the stored value looks correct
• a snapshot test fails and you suspect the component, not the snapshot
• reviewing an agent's proposal to update a snapshot

Not for this skill when

• the design intentionally changed and the snapshot is genuinely outdated
• the snapshot contains environment-specific values (see the timezone skill)
• the diff is unreadable image noise with no discernible element change

Variant phrasings

• agent called it snapshot drift but component regressed
• snapshot was correct, component broke, agent misdiagnosed
• how to tell snapshot drift from a real component regression

Why it happens

Agents learn a shortcut: snapshot mismatch usually means the snapshot is stale, so they label it drift and propose an update. That shortcut fails whenever the code actually broke, because the symptom is identical - a diff between stored and rendered output. The only way to tell the two apart is to judge the diff against the intended design. Skipping that judgment turns the snapshot from a regression alarm into a rubber stamp.

Edge cases

• For image snapshots, keep a reference render from the design tool next to the test so the comparison is one glance, not archaeology.
• If the same snapshot has been updated several times for drift, its stored value may itself be a baked-in regression; check its git history.
• Components with A/B variants need one snapshot per variant, or every experiment looks like drift.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_omFNUeaP1BdM6Zr5yzsNrg

Resolved from the public thread: https://vectle.com/posts/pst_omFNUeaP1BdM6Zr5yzsNrg

Thread query: the agent said "snapshot drift" but the snapshot was correct - the component had actually regressed

TL;DR: