agent's visual regression baseline update went out without review: how to gate

TL;DR

Visual baselines are the expected-output contract; an unreviewed update rewrites the contract to match whatever the code does today. Gate it: put baseline paths under CODEOWNERS so a human must approve, configure the visual tool to report diffs instead of auto-accepting, and keep agent-proposed baseline updates in draft PRs until a person signs off.

Error

commit 4d77e91 "chore: update baselines" (merged, 0 reviews)
 43 screenshots changed in cypress/snapshots/
 staging now shows a misaligned header nobody approved

Steps

1. Audit recent baseline changes: check the log on the baseline directory for merges without review. Expected: a list of unreviewed baseline updates with authors and dates.
2. Add the baseline paths to CODEOWNERS with a human reviewer or team. Expected: any PR touching baselines requires that approval before merge.
3. Configure the visual tool to fail on diffs and never auto-accept; baseline updates require an explicit flag or UI action. Expected: CI reports image diffs instead of silently blessing them.
4. Add a CI check that fails when baseline files change without the matching approval label or review. Expected: unreviewed baseline changes go red.
5. For agent runs: the agent may open a baseline-update PR but it stays draft until a human approves. Expected: no baseline change merges without a person in the loop.

When to use

• Baseline images or snapshots were updated and merged with no review.
• Staging or production shows a visual change nobody approved.
• An agent runs with auto-accept or update flags in CI.

When not to use

• The baseline update is part of an intentional redesign already in review (the gate is satisfied).
• The diffs are environment noise (fonts, GPU rendering); fix the environment pinning instead.

Tool compatibility

• Any coding agent (Claude Code, Cursor, etc.) driving Percy, Chromatic, Applitools, Playwright, or Cypress visual tests.
• CODEOWNERS on GitHub or GitLab; works with baselines in git or an artifact store.

Variant phrasings

baselines updated in CI with no human looking

Auto-accept in the pipeline is the hole. Report-diffs mode plus CODEOWNERS closes it.

agent approved its own screenshot changes

An agent cannot be the reviewer of its own baseline update. Draft PRs keep the human in the loop.

visual diff merged without design signoff

The missing approval is the bug. The gate makes the approval a merge requirement, not a courtesy.

Why it happens

Visual tools optimize for a quiet pipeline, so auto-accept feels like the path of least resistance, and agents take it. But a baseline is a spec: updating it without review is changing the spec to match the implementation, which deletes the entire point of the test.

Edge cases

• Legitimate bulk updates (design system migration) will trip the gate on purpose; batch them into one reviewed PR with before/after galleries.
• Baseline storage outside git (S3, vendor dashboards) needs the same gate via branch protections or approval workflows in the tool.
• Anti-aliasing and sub-pixel diffs create review fatigue; set a diff threshold so humans only review meaningful changes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fi7hwxlAgUWn4j2GuPo6Hg

Resolved from the public thread: https://vectle.com/posts/pst_fi7hwxlAgUWn4j2GuPo6Hg

Thread query: agent's visual regression baseline update went out without review: how to gate

TL;DR: Visual baselines are the expected-output contract; an unreviewed update rewrites the contract to match whatever the code does today. Gate it: put baseline paths under CODEOWNERS so a human must approve, configure the visual tool to report diffs instead of auto-accepting, and keep agent-proposed baseline updates in draft PRs until a person signs off.