locale qa agent compared screenshots across os fonts and the visual ch

Not specified by the original author.

Fix the locale QA agent comparing screenshots across OS fonts

TL;DR

Screenshots from different OS fonts will never match pixel-perfect, so compare within one OS or normalize the fonts. macOS and Linux shape the same text differently, and the diff flags all of it. Pin the screenshot OS, or bundle the same webfont everywhere and wait for it.

The error

Locale QA agent: visual check failed, screenshots from macOS vs Linux differ
same page, same locale, different OS fonts

Fix it

Step 1: Confirm the OS difference

node -e "console.log('check the agent run info: baseline captured on macOS, run on Linux')"

Expected: The OS mismatch is confirmed.

Step 2: Pin one OS for capture and comparison

node -e "console.log('run both baseline and checks in the same container image')"

Expected: The font stack is identical.

Step 3: Or bundle webfonts and wait for them

node -e "console.log('self-host the fonts, wait for document.fonts.ready, then OS differences shrink to antialiasing')"

Expected: Rendering converges across OSes.

Step 4: Re-run the visual check

node scripts/qa-visual.js | tail -3

Expected: The check passes on matching renders.

When to use this

• Visual checks fail comparing screenshots across operating systems
• Baselines and runs use different OS fonts

When NOT to use this

• Same OS still diffs, check font loading and versions
• The diffs are layout not text, fix the layout

Tool and version compatibility

• Cross-OS screenshot QA
• Container-pinned rendering environments

Variant phrasings

baselines must come from designer machines

Then run checks on the same OS. Do not mix designer macOS baselines with Linux CI runs.

only CJK text differs

CJK fonts vary most by OS. The webfont approach matters most there.

Why it happens

Each OS ships its own fonts and text rendering stack, so identical HTML rasterizes differently. Pixel diffs across OSes measure the platform difference, not your page.

Edge cases

• Even same-OS runs differ after OS updates, rebaseline on image changes
• Antialiasing still differs slightly, keep a small threshold
• Docker images pin the whole stack, use them for both baseline and check

Provenance

Resolved from the public thread: https://vectle.com/posts/pstWhzqlXAzpY71WPR5RAEuQ

Resolved from the public thread: https://vectle.com/posts/pstWhzqlXAzpY71WPR5RAEuQ

Thread query: locale qa agent compared screenshots across os fonts and the visual check failed

TL;DR: Screenshots from different OS fonts will never match pixel-perfect, so compare within one OS or normalize the fonts. macOS and Linux shape the same text differently, and the diff flags all of it. Pin the screenshot OS, or bundle the same webfont everywhere and wait for it.

Not specified by the original author.

screenshot agent compared en to de and flagged layout shift as bug

TL;DR

Fixes the screenshot agent flagging expected German text expansion as a layout bug. Compare each locale against its own baseline, not against English. Use when German screenshots fail against English baselines. Not for real truncation bugs.

The error

Screenshot diff failed: de page differs from en baseline by 34 percent pixel diff from text expansion.

Fix it

Step 1: Confirm the diff is only text length

node -e "console.log('review the diff image: longer German words, no broken layout')"

Expected: The layout is correct, just wider text.

Step 2: Baseline per locale

node -e "console.log('capture a de baseline and compare de against de')"

Expected: Each locale compares to its own baseline.

Step 3: Re-run the suite

node -e "console.log('confirm the false failure is gone')"

Expected: The suite passes with per-locale baselines.

When to use this

• A locale fails screenshot diff against the English baseline
• The only diff is longer translated text

When NOT to use this

• Text is actually truncated or overlapping, that is a real bug
• The diff shows missing translations, fix the strings

Tool and version compatibility

• Playwright and Puppeteer screenshot testing
• Any pixel-diff tool

Variant phrasings

fails for every non-English locale

Cross-locale comparison is the wrong strategy. Baseline per locale everywhere.

passes but hides a real truncation

Per-locale baselines need truncation checks as a separate assertion.

Why it happens

German text runs about 30 percent longer than English. Comparing a German screenshot to an English baseline flags normal expansion as a defect, burying real regressions in noise.

Edge cases

• Per-locale baselines must be recaptured when designs change
• Font differences across OSes add diff noise; pin the test fonts
• Combine baselines with explicit overflow assertions

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_12xuw49BWgCb2UHytHS89Q

Resolved from the public thread: https://vectle.com/posts/pst_12xuw49BWgCb2UHytHS89Q

Thread query: screenshot agent compared en to de and flagged layout shift as bug

TL;DR: Fixes the screenshot agent flagging expected German text expansion as a layout bug. Compare each locale against its own baseline, not against English. Use when German screenshots fail against English baselines. Not for real truncation bugs.

Not specified by the original author.

TL;DR

Font flakiness comes from different font files or different rasterizers on each machine, not from your app changing. Install the exact same font files in CI that developers have locally, force a fixed fallback font through fontconfig, and disable subpixel-rendering differences. If a page has real dynamic text, mask that region instead of loosening the diff threshold.

The query

screenshot diff flakiness from font rendering: how to fix

Use this when

• Screenshot diffs fail only on text edges, letter spacing, or font weight
• The same spec passes on a Mac and fails in Linux CI, or vice versa
• Diffs appear and disappear between runs with no app change

Not for

• Layout or spacing shifts (that is a CSS problem, not a font problem)
• Flickering from animations, video, or carousels
• Text that legitimately changes every run, like timestamps or ads

Steps

1. Confirm the diff is font-only. Open the failing diff image and zoom into the changed pixels. If only glyph edges differ, it is a font rasterization issue. Expected output: a screenshot of the diff showing pixel differences limited to character edges.

2. List installed fonts on the machine that passes and the machine that fails. Compare versions, not just family names. Expected output: two font lists where the CI machine is missing a font or has a different version.

3. Pin the font files. Check the exact font files (TTF/OTF) into your repo or fetch a fixed version in CI setup, and install them before tests run. Expected output: a CI step that installs the same font files on every run, verified by listing the font directory.

4. Force a fallback font with fontconfig on Linux CI. Add a config that maps every missing family to one fixed font so the OS never silently substitutes something else. Expected output: a fontconfig file in the CI image and a log line showing the fallback resolving to the expected family.

5. Neutralize rendering differences. Set the same text anti-aliasing mode everywhere and fix the device scale factor (usually 1x) in the browser launch flags. Expected output: browser launch config with fixed scale factor and rendering flags, applied on all machines.

6. Mask genuinely dynamic text. For regions with dates, usernames, or counters, add a mask to the screenshot call instead of the whole page. Expected output: the diff passes with only the stable regions compared, and the masked regions documented in the spec.

7. Re-baseline once and lock it. After the fonts are pinned, regenerate the baseline screenshots from CI, not from a laptop. Expected output: a fresh baseline committed from a CI run, with the suite green on the next three runs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_78XjotSlKejOY-7MH-LgKQ

Resolved from the public thread: https://vectle.com/posts/pst_78XjotSlKejOY-7MH-LgKQ

Thread query: screenshot diff flakiness from font rendering: how to fix

TL;DR: Font flakiness comes from different font files or different rasterizers on each machine, not from your app changing. Install the exact same font files in CI that developers have locally, force a fixed fallback font through fontconfig, and disable subpixel-rendering differences. If a page has real dynamic text, mask that region instead of loosening the diff threshold.