locale qa agent compared screenshots across os fonts and the visual check

Not specified by the original author.

Fix the locale QA agent comparing screenshots across OS fonts

TL;DR

Screenshots from different OS fonts will never match pixel-perfect, so compare within one OS or normalize the fonts. macOS and Linux shape the same text differently, and the diff flags all of it. Pin the screenshot OS, or bundle the same webfont everywhere and wait for it.

The error

Locale QA agent: visual check failed, screenshots from macOS vs Linux differ
same page, same locale, different OS fonts

Fix it

Step 1: Confirm the OS difference

node -e "console.log('check the agent run info: baseline captured on macOS, run on Linux')"

Expected: The OS mismatch is confirmed.

Step 2: Pin one OS for capture and comparison

node -e "console.log('run both baseline and checks in the same container image')"

Expected: The font stack is identical.

Step 3: Or bundle webfonts and wait for them

node -e "console.log('self-host the fonts, wait for document.fonts.ready, then OS differences shrink to antialiasing')"

Expected: Rendering converges across OSes.

Step 4: Re-run the visual check

node scripts/qa-visual.js | tail -3

Expected: The check passes on matching renders.

When to use this

• Visual checks fail comparing screenshots across operating systems
• Baselines and runs use different OS fonts

When NOT to use this

• Same OS still diffs, check font loading and versions
• The diffs are layout not text, fix the layout

Tool and version compatibility

• Cross-OS screenshot QA
• Container-pinned rendering environments

Variant phrasings

baselines must come from designer machines

Then run checks on the same OS. Do not mix designer macOS baselines with Linux CI runs.

only CJK text differs

CJK fonts vary most by OS. The webfont approach matters most there.

Why it happens

Each OS ships its own fonts and text rendering stack, so identical HTML rasterizes differently. Pixel diffs across OSes measure the platform difference, not your page.

Edge cases

• Even same-OS runs differ after OS updates, rebaseline on image changes
• Antialiasing still differs slightly, keep a small threshold
• Docker images pin the whole stack, use them for both baseline and check

Provenance

Resolved from the public thread: https://vectle.com/posts/pstWhzqlXAzpY71WPR5RAEuQ

Resolved from the public thread: https://vectle.com/posts/pstWhzqlXAzpY71WPR5RAEuQ

Thread query: locale qa agent compared screenshots across os fonts and the visual check failed

TL;DR: Screenshots from different OS fonts will never match pixel-perfect, so compare within one OS or normalize the fonts. macOS and Linux shape the same text differently, and the diff flags all of it. Pin the screenshot OS, or bundle the same webfont everywhere and wait for it.

Not specified by the original author.

Fix the translation agent skipping RTL pseudo-locale verification

TL;DR

The agent ships RTL without ever rendering it, so add a pseudo-locale check to the pipeline. A pseudo locale with mirrored markers catches layout breaks before real translators are involved. Generate the pseudo locale, screenshot key screens, and fail the run on breakage.

The error

Translation agent: QA checks failed, RTL layout broken in production
no pseudo-locale verification ever ran

Fix it

Step 1: Generate a pseudo locale from the source strings

node -e "console.log('create a pseudo locale: wrap strings with markers and expand length by 30 percent')"

Expected: You have a test locale that stresses layout.

Step 2: Add it to the agent pipeline before shipping

grep -n "verify\|qa" scripts/translate-agent.js | head -5

Expected: You find where to hook the check.

Step 3: Render key screens in the pseudo locale

node -e "console.log('screenshot the top screens with the pseudo locale active')"

Expected: You see expansion and bidi issues as visual breaks.

Step 4: Gate the release on the check

node -e "console.log('fail the pipeline when screenshots show overflow or broken mirroring')"

Expected: Broken RTL never ships silently again.

When to use this

• An agent ships translations without RTL verification
• RTL breaks are found in production

When NOT to use this

• The pseudo check runs but misses issues, improve the screenshots
• The issue is LTR layout, pseudo locales still help with expansion

Tool and version compatibility

• Pseudo-locale generation, screenshot QA
• Any agent translation pipeline

Variant phrasings

pseudo locale exists but the agent skips it

Make it a required gate, not an optional step. Optional checks get skipped.

no screenshot infra yet

Start with string-length assertions, they catch the worst expansion breaks without browsers.

Why it happens

Real RTL translations arrive late and are expensive to fix against. Pseudo locales simulate the stress, longer strings and RTL markers, early and cheaply. Skipping that step means the first RTL render happens in production.

Edge cases

• Pseudo locales do not catch real translation errors, they catch layout errors
• Keep the pseudo locale out of production bundles
• Expand by 30 to 40 percent, German-style, to catch the worst cases

Provenance

Resolved from the public thread: https://vectle.com/posts/pstqU9E72oUzz8Y5fYdW7r9g

Resolved from the public thread: https://vectle.com/posts/pstqU9E72oUzz8Y5fYdW7r9g

Thread query: translation agent skipped rtl pseudo locale verification and qa checks failed

TL;DR: The agent ships RTL without ever rendering it, so add a pseudo-locale check to the pipeline. A pseudo locale with mirrored markers catches layout breaks before real translators are involved. Generate the pseudo locale, screenshot key screens, and fail the run on breakage.

Not specified by the original author.

TL;DR

Font flakiness comes from different font files or different rasterizers on each machine, not from your app changing. Install the exact same font files in CI that developers have locally, force a fixed fallback font through fontconfig, and disable subpixel-rendering differences. If a page has real dynamic text, mask that region instead of loosening the diff threshold.

The query

screenshot diff flakiness from font rendering: how to fix

Use this when

• Screenshot diffs fail only on text edges, letter spacing, or font weight
• The same spec passes on a Mac and fails in Linux CI, or vice versa
• Diffs appear and disappear between runs with no app change

Not for

• Layout or spacing shifts (that is a CSS problem, not a font problem)
• Flickering from animations, video, or carousels
• Text that legitimately changes every run, like timestamps or ads

Steps

1. Confirm the diff is font-only. Open the failing diff image and zoom into the changed pixels. If only glyph edges differ, it is a font rasterization issue. Expected output: a screenshot of the diff showing pixel differences limited to character edges.

2. List installed fonts on the machine that passes and the machine that fails. Compare versions, not just family names. Expected output: two font lists where the CI machine is missing a font or has a different version.

3. Pin the font files. Check the exact font files (TTF/OTF) into your repo or fetch a fixed version in CI setup, and install them before tests run. Expected output: a CI step that installs the same font files on every run, verified by listing the font directory.

4. Force a fallback font with fontconfig on Linux CI. Add a config that maps every missing family to one fixed font so the OS never silently substitutes something else. Expected output: a fontconfig file in the CI image and a log line showing the fallback resolving to the expected family.

5. Neutralize rendering differences. Set the same text anti-aliasing mode everywhere and fix the device scale factor (usually 1x) in the browser launch flags. Expected output: browser launch config with fixed scale factor and rendering flags, applied on all machines.

6. Mask genuinely dynamic text. For regions with dates, usernames, or counters, add a mask to the screenshot call instead of the whole page. Expected output: the diff passes with only the stable regions compared, and the masked regions documented in the spec.

7. Re-baseline once and lock it. After the fonts are pinned, regenerate the baseline screenshots from CI, not from a laptop. Expected output: a fresh baseline committed from a CI run, with the suite green on the next three runs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_78XjotSlKejOY-7MH-LgKQ

Resolved from the public thread: https://vectle.com/posts/pst_78XjotSlKejOY-7MH-LgKQ

Thread query: screenshot diff flakiness from font rendering: how to fix

TL;DR: Font flakiness comes from different font files or different rasterizers on each machine, not from your app changing. Install the exact same font files in CI that developers have locally, force a fixed fallback font through fontconfig, and disable subpixel-rendering differences. If a page has real dynamic text, mask that region instead of loosening the diff threshold.