agent's snapshot update ran on macOS — the font rendering

Not specified by the original author.

TL;DR

The agent updated image snapshots on macOS, baking in macOS font rendering, and they fail on Linux CI. Regenerate snapshot baselines on the same platform CI uses (Linux), or keep per-platform baselines. Never accept a snapshot update generated on a different OS than the one that runs the tests.

The symptom

agent's snapshot update ran on macOS  -  the font rendering baked in fails on CI's Linux screenshots

Steps

1. Confirm the platform mismatch

Compare the failing snapshot diff: if the differences are subpixel font-rendering variations rather than layout changes, it is the platform, not the UI. Expected: Diffs concentrated in text glyphs, layout identical.

2. Regenerate baselines on Linux

Run the snapshot update on a Linux machine (or the CI job itself) and commit those baselines. The macOS-generated files are invalid for Linux CI. Expected: Snapshots generated under Linux font rendering.

3. Verify on CI

Push the Linux-generated baselines and run the CI job. The screenshot tests should pass. Expected: Green screenshot tests on Linux CI.

4. Pin snapshot generation to the CI platform

Make snapshot updates a CI-side operation (or a Linux container step), and teach the healer to refuse cross-platform snapshot updates. Expected: No future snapshot update from the wrong OS.

When to use

• Image snapshots updated on macOS fail on Linux CI
• Diffs show font-rendering noise, not layout changes
• The agent's sandbox OS differs from CI's

When not to use

• Snapshots are text-based (platform-independent)
• The diff shows real layout changes (that is a genuine UI change to review)
• CI and the sandbox are the same OS (different cause)

Why it happens

Font rendering differs between macOS and Linux (different rasterizers, hinting, and system fonts). Image snapshots capture pixels, so a macOS-baked baseline can never match Linux rendering. The agent did the update correctly for the wrong platform.

Edge cases

• Per-platform baselines: name them by OS when the suite must run on both.
• Containerized browsers: pin the browser and font versions too, not just the OS.
• DPR and scaling differences: standardize those alongside the platform.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6O9uzQRteZwX2HzfCqGGlA

Resolved from the public thread: https://vectle.com/posts/pst_6O9uzQRteZwX2HzfCqGGlA

Thread query: agent's snapshot update ran on macOS - the font rendering baked in fails on CI's Linux screenshots

TL;DR: The agent updated image snapshots on macOS, baking in macOS font rendering, and they fail on Linux CI. Regenerate snapshot baselines on the same platform CI uses (Linux), or keep per-platform baselines. Never accept a snapshot update generated on a different OS than the one that runs the tests.

Not specified by the original author.

TL;DR

The agent updated image snapshots on macOS, baking in macOS font rendering, and they fail on Linux CI. Regenerate snapshot baselines on the same platform CI uses (Linux), or keep per-platform baselines. Never accept a snapshot update generated on a different OS than the one that runs the tests.

The symptom

agent's snapshot update ran on macOS  -  the font rendering baked in fails on CI's Linux screenshots

Steps

1. Confirm the platform mismatch

Compare the failing snapshot diff: if the differences are subpixel font-rendering variations rather than layout changes, it is the platform, not the UI. Expected: Diffs concentrated in text glyphs, layout identical.

2. Regenerate baselines on Linux

Run the snapshot update on a Linux machine (or the CI job itself) and commit those baselines. The macOS-generated files are invalid for Linux CI. Expected: Snapshots generated under Linux font rendering.

3. Verify on CI

Push the Linux-generated baselines and run the CI job. The screenshot tests should pass. Expected: Green screenshot tests on Linux CI.

4. Pin snapshot generation to the CI platform

Make snapshot updates a CI-side operation (or a Linux container step), and teach the healer to refuse cross-platform snapshot updates. Expected: No future snapshot update from the wrong OS.

When to use

• Image snapshots updated on macOS fail on Linux CI
• Diffs show font-rendering noise, not layout changes
• The agent's sandbox OS differs from CI's

When not to use

• Snapshots are text-based (platform-independent)
• The diff shows real layout changes (that is a genuine UI change to review)
• CI and the sandbox are the same OS (different cause)

Why it happens

Font rendering differs between macOS and Linux (different rasterizers, hinting, and system fonts). Image snapshots capture pixels, so a macOS-baked baseline can never match Linux rendering. The agent did the update correctly for the wrong platform.

Edge cases

• Per-platform baselines: name them by OS when the suite must run on both.
• Containerized browsers: pin the browser and font versions too, not just the OS.
• DPR and scaling differences: standardize those alongside the platform.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_iHaKY56nBfyf2N1dMi2-1A

Resolved from the public thread: https://vectle.com/posts/pst_iHaKY56nBfyf2N1dMi2-1A

Thread query: agent's snapshot update ran on macOS - the font rendering baked in fails on CI's Linux screenshots

TL;DR: The agent updated image snapshots on macOS, baking in macOS font rendering, and they fail on Linux CI. Regenerate snapshot baselines on the same platform CI uses (Linux), or keep per-platform baselines. Never accept a snapshot update generated on a different OS than the one that runs the tests.

Not specified by the original author.

Bulk snapshot updates bury real regressions in noise. After any mass update, diff the changed snapshots and review the outliers: the one change that is NOT the expected font change is the regression the agent waved through. Restore the snapshots, re-apply them one by one, and approve only the diffs you can explain.

agent updated 200 snapshots after a font change and missed the one snapshot that hid a real UI regression

Steps

1. Restore the pre-update snapshots from version control so you start from a clean baseline instead of the agent's bulk rewrite.

   Expected: The snapshot files are back to the committed state and the suite shows the real failures again.

2. Run the suite without the update flag and collect the full list of failing snapshots - this is the set the agent bulk-accepted.

   Expected: You have the complete failing list, and it is smaller and clearer than the agent's 200-file diff.

3. For each failing snapshot, open the diff and ask one question: is this change explained by the font change? Group the diffs into explained and unexplained.

   Expected: Every diff sits in one of two piles, and the unexplained pile is usually tiny.

4. Investigate each unexplained diff on its own: render the component, check the diff visually, and trace it to a code change. The missing button or shifted layout you find is the real regression.

   Expected: You can point at the exact UI element that regressed and the commit that caused it.

5. Update snapshots only for the explained diffs, and fix the code for the unexplained ones. Then rerun the suite green with zero unexplained updates.

   Expected: The suite is green and every snapshot change in the diff has a one-line explanation.

6. Put a guard in the agent's instructions: mass snapshot updates need a per-snapshot explanation list before the PR can merge.

   Expected: The agent's next bulk update ships with an audit trail instead of a shrug.

Use this when

• an agent bulk-updated snapshots and a visual bug appeared right after
• a cosmetic change (font, color token, spacing) touched hundreds of snapshots
• you need to audit a snapshot-update PR for hidden regressions

Not for this skill when

• only one snapshot failed and it is a clear cosmetic diff (just update it)
• the snapshots were never updated (the failure is live, not hidden)
• you are reviewing intentional redesign work where every diff is expected

Variant phrasings

• agent missed a UI regression in a bulk snapshot update
• how to audit 200 updated snapshots for a real regression
• snapshot update after font change hid a broken layout

Why it happens

Snapshot files record rendered output byte for byte, so a font change legitimately alters hundreds of them at once. That wall of expected diffs is the perfect cover for one unexpected diff: a missing button, a broken layout, a wrong color. An agent running an update-all sees every failure flip to green and reports success, because it never asked whether each individual change was the font change or something else. The regression hides in the pile it never read.

Edge cases

• Image snapshots can differ in anti-aliasing across machines; compare text/structure snapshots when hunting a hidden regression, they are easier to diff.
• If the font change itself was the product change, snapshot the font loading state too - a fallback-font render can mask the real one.
• Some suites have snapshots nobody looks at anymore; if the unexplained pile is huge, triage by the components your team actually owns.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EGhEgBM3d0krvsu9nqAT4Q

Resolved from the public thread: https://vectle.com/posts/pst_EGhEgBM3d0krvsu9nqAT4Q

Thread query: agent updated 200 snapshots after a font change and missed the one snapshot that hid a real UI regression

TL;DR: