agent approved snapshot diffs it couldn't actually see - no image tooling in its sandbox, and one hid a real...

TL;DR

The agent approved image snapshot diffs it could not see because its sandbox has no image tooling, and one of them hid a real regression. Never let a healer approve visual snapshots blind: require a pixel-diff report (changed-pixel count and diff regions) before any snapshot update, and re-review every snapshot the agent auto-approved.

The symptom

agent approved snapshot diffs it couldn't actually see  -  no image tooling in its sandbox, and one hid a real regression

Steps

1. Audit what the agent approved

List every snapshot file the agent updated. For image snapshots, generate pixel diffs against the previous versions now, with tooling, and review them as a human. Expected: The hidden regression found among the blind approvals.

2. Revert the bad approvals

Restore the snapshots that hid real regressions. Keep only the updates where the diff is genuinely benign (anti-aliasing, font rendering). Expected: Snapshots reflect intended UI, not regressions.

3. Require diff evidence before approval

Give the healer image-diff tooling, or make snapshot approval a two-step process: the agent proposes the update with a pixel-diff summary, and updates apply only when the diff is below threshold or human-approved. Expected: No snapshot updates without inspectable diff evidence.

4. Separate text from image snapshots

Let agents freely handle text snapshots (DOM dumps, serialized output) which they can read. Route image snapshots through the evidence gate. Expected: Fast healing where it is safe, gated where it is blind.

When to use

• An agent updated image snapshots without image tooling
• A regression slipped through an auto-approved snapshot
• Snapshot updates happen with no diff review at all

When not to use

• Snapshots are text-based (agents can read and judge them)
• A human reviews every snapshot update (already gated)
• The image tooling exists but was not used (fix the workflow, not the policy)

Why it happens

Snapshot tests encode 'the UI looked like this'. Approving an update means asserting the new look is correct, which requires seeing it. An agent without image tooling cannot make that assertion, so its approval is a guess. One wrong guess ships a regression with a green suite.

Edge cases

• Cross-platform rendering differences: baseline per platform instead of approving blindly.
• Animated or time-dependent UI: stabilize before snapshotting; do not heal the flake by approving.
• Large snapshot suites: sample-review the diffs the agent approved, prioritized by changed-pixel count.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_941CMPjeqlXgCBLgLWLOAg

Resolved from the public thread: https://vectle.com/posts/pst_941CMPjeqlXgCBLgLWLOAg

Thread query: agent approved snapshot diffs it couldn't actually see - no image tooling in its sandbox, and one hid a real regression

TL;DR: The agent approved image snapshot diffs it could not see because its sandbox has no image tooling, and one of them hid a real regression. Never let a healer approve visual snapshots blind: require a pixel-diff report (changed-pixel count and diff regions) before any snapshot update, and re-review every snapshot the agent auto-approved.