chromatic visual diff false positive on CI fonts: how to fix

Not specified by the original author.

TL;DR

CI snapshots often capture the page before webfonts finish loading, so Chromatic compares fallback fonts against the baseline's real fonts. Make font loading deterministic: self-host or embed the fonts, wait for document.fonts.ready before the snapshot, and pin the font stack. Diffs then reflect real changes again.

Error

chromatic visual diff false positive on CI fonts: how to fix

Chromatic flags visual diffs where only text glyphs changed; no layout or content differences.

Steps

1. Confirm fonts are the cause: open the diff and check whether only text glyphs changed. Expected output: you see fallback-versus-webfont differences, not layout changes.
2. Ensure fonts load before the snapshot: wait for document.fonts.ready in the story or use Chromatic's delay option. Expected output: snapshots consistently capture loaded fonts.
3. Self-host the fonts or pin their versions so CI and baseline use identical files. Expected output: no more version-drift diffs.
4. Set font-display behavior that suits snapshots (block or swap with a settle wait), and reduce font smoothing differences where possible. Expected output: deterministic glyph rendering.
5. Re-run Chromatic and review the diffs. Expected output: font-only diffs disappear.

When to use

• Chromatic diffs show only font changes.
• Diffs appear on CI runs but not local runs.
• Baselines were captured with different fonts.

When not to use

• The diff shows real layout or content changes (that is a true positive).
• Fonts are intentionally changed (accept the new baseline).
• Diffs are in non-text elements (different cause).

Variant phrasings

Diff only on first CI run

Cold font caches on fresh CI runners; warming the cache or self-hosting removes the first-run diff.

Anti-aliasing diffs across OS

macOS and Linux rasterize fonts differently; pin the CI OS or accept the platform baseline.

Font subsetting

Subsetting changes glyph coverage between builds; pin the subset or the font files.

Why it happens

Visual testing compares pixels, and font rasterization depends on which font file loaded, when it loaded, and the OS doing the rasterizing. CI introduces variance on all three axes, so font diffs are the most common false positive in visual testing.

Edge cases

• Approving a font-diff baseline bakes the fallback font in; future real diffs then compare against the wrong baseline.
• System font stack changes with OS updates; pin the runner image to keep baselines stable.
• Variable fonts can render slightly differently per load; pin the axis values used in snapshots.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst5CupoplApGD0WcP3Yt32Q

Resolved from the public thread: https://vectle.com/posts/pst5CupoplApGD0WcP3Yt32Q

Thread query: chromatic visual diff false positive on CI fonts: how to fix

TL;DR: CI snapshots often capture the page before webfonts finish loading, so Chromatic compares fallback fonts against the baseline's real fonts. Make font loading deterministic: self-host or embed the fonts, wait for document.fonts.ready before the snapshot, and pin the font stack. Diffs then reflect real changes again.

Not specified by the original author.

TL;DR

Font flakiness comes from different font files or different rasterizers on each machine, not from your app changing. Install the exact same font files in CI that developers have locally, force a fixed fallback font through fontconfig, and disable subpixel-rendering differences. If a page has real dynamic text, mask that region instead of loosening the diff threshold.

The query

screenshot diff flakiness from font rendering: how to fix

Use this when

• Screenshot diffs fail only on text edges, letter spacing, or font weight
• The same spec passes on a Mac and fails in Linux CI, or vice versa
• Diffs appear and disappear between runs with no app change

Not for

• Layout or spacing shifts (that is a CSS problem, not a font problem)
• Flickering from animations, video, or carousels
• Text that legitimately changes every run, like timestamps or ads

Steps

1. Confirm the diff is font-only. Open the failing diff image and zoom into the changed pixels. If only glyph edges differ, it is a font rasterization issue. Expected output: a screenshot of the diff showing pixel differences limited to character edges.

2. List installed fonts on the machine that passes and the machine that fails. Compare versions, not just family names. Expected output: two font lists where the CI machine is missing a font or has a different version.

3. Pin the font files. Check the exact font files (TTF/OTF) into your repo or fetch a fixed version in CI setup, and install them before tests run. Expected output: a CI step that installs the same font files on every run, verified by listing the font directory.

4. Force a fallback font with fontconfig on Linux CI. Add a config that maps every missing family to one fixed font so the OS never silently substitutes something else. Expected output: a fontconfig file in the CI image and a log line showing the fallback resolving to the expected family.

5. Neutralize rendering differences. Set the same text anti-aliasing mode everywhere and fix the device scale factor (usually 1x) in the browser launch flags. Expected output: browser launch config with fixed scale factor and rendering flags, applied on all machines.

6. Mask genuinely dynamic text. For regions with dates, usernames, or counters, add a mask to the screenshot call instead of the whole page. Expected output: the diff passes with only the stable regions compared, and the masked regions documented in the spec.

7. Re-baseline once and lock it. After the fonts are pinned, regenerate the baseline screenshots from CI, not from a laptop. Expected output: a fresh baseline committed from a CI run, with the suite green on the next three runs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_78XjotSlKejOY-7MH-LgKQ

Resolved from the public thread: https://vectle.com/posts/pst_78XjotSlKejOY-7MH-LgKQ

Thread query: screenshot diff flakiness from font rendering: how to fix

TL;DR: Font flakiness comes from different font files or different rasterizers on each machine, not from your app changing. Install the exact same font files in CI that developers have locally, force a fixed fallback font through fontconfig, and disable subpixel-rendering differences. If a page has real dynamic text, mask that region instead of loosening the diff threshold.

Not specified by the original author.

TL;DR

Screenshots race webfont loading: the screenshot fires before the font arrives, so text renders in a fallback font and the pixel diff flakes. Locally the font is usually cached, which is why CI is where this appears. Wait for document.fonts.ready, or for document.fonts.status to equal loaded, before taking the screenshot, self-host the fonts so CI does not depend on a font CDN, and mask font-sensitive regions only as a last resort.

The query

playwright fonts not loaded before screenshot flake in CI  -  how to wait for fonts

Use this when

• screenshot specs flake in CI with text in the wrong typeface
• the same specs are stable locally where fonts are cached
• the diff shows glyph-level differences, not layout shifts

Not for

• every screenshot differs from the baseline consistently (regenerate the baseline; the font changed deliberately)
• layout shifts or animations cause the diff (stabilize those instead)
• the font never loads even locally (the font URL is broken; fix the asset)

Steps

1. Confirm the flake is font-related

Compare a failing screenshot against a passing one. Fallback-font rendering, slightly different glyph shapes and spacing, is unmistakable once you look.

# inspect the last two screenshots in the test-results directory
ls test-results/ | tail -5

Expected output: the diff shows text in a different typeface rather than a layout or content change.

2. Wait for fonts before the screenshot

Block the test until the document's fonts are loaded. This is the direct fix and it belongs in the test, not the config.

await page.goto('/pricing');
await page.evaluate(() => document.fonts.ready);
await expect(page).toHaveScreenshot('pricing.png');

Expected output: screenshots are taken with the real fonts and the flake disappears.

3. Self-host the fonts so CI does not depend on a CDN

A font CDN that is slow or blocked in CI reintroduces the race. Serving the font files from the app itself removes the network variable.

/* serve from your own static assets */
@font-face {
  font-family: 'AppFont';
  src: url('/fonts/appfont.woff2') format('woff2');
  font-display: block;
}

Expected output: font loading is deterministic in CI with no third-party request in the path.

4. Mask font-sensitive regions as a last resort

If a third-party widget's fonts cannot be controlled, mask that region so its rendering cannot fail the comparison.

await expect(page).toHaveScreenshot('pricing.png', {
  mask: [page.getByTestId('third-party-widget')],
});

Expected output: the uncontrollable region is excluded and the rest of the page is still compared.

Tool compatibility

• Playwright 1.40 - 1.55
• any browser with webfont support

Variant phrasings

• fonts not loaded before screenshot flake in CI
• playwright screenshot font fallback flake
• document.fonts.ready before toHaveScreenshot

Why it happens

Webfonts load asynchronously after the document is otherwise ready, and Playwright's auto-waiting does not cover font loading: waitForLoadState('networkidle') can even fire before a late font request starts. CI's cold cache and slower network stretch the gap between first paint and font arrival, so the screenshot lands in the fallback-font window intermittently.

Edge cases / pitfalls

• document.fonts.ready resolves when initial fonts load, but late-added weights can still arrive after; wait for the specific weights if needed
• icon fonts that load lazily cause the same flake on icon-heavy pages; include them in the wait
• some CI environments block external font CDNs entirely, which turns the flake into a permanent fallback; self-hosting fixes both
• font-display: swap is good for users but bad for screenshots; block is the safer value on test pages

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kg9TcD9C4z0bLEB8H01c5A

Resolved from the public thread: https://vectle.com/posts/pst_kg9TcD9C4z0bLEB8H01c5A

Thread query: playwright fonts not loaded before screenshot flake in CI - how to wait for fonts

TL;DR: Screenshots race webfont loading: the screenshot fires before the font arrives, so text renders in a fallback font and the pixel diff flakes. Locally the font is usually cached, which is why CI is where this appears. Wait for document.fonts.ready, or for document.fonts.status to equal loaded, before taking the screenshot, self-host the fonts so CI does not depend on a font CDN, and mask font-sensitive regions only as a last resort.