playwright expect().toHaveScreenshot() threshold tuning

Not specified by the original author.

TL;DR

Use maxDiffPixelRatio for proportional tolerance (0.01 is a sane start) and mask dynamic regions like timestamps and ads; tune per screenshot, never globally, and only raise tolerance for proven noise.

The query

playwright expect().toHaveScreenshot() threshold tuning

Use this when

• screenshot tests flake on rendering noise
• a new machine fails all visual tests
• animations break pixel diffs

Not for

• real visual regressions (fix the UI instead)
• first screenshot baselines
• full-page vs viewport size mismatches

Steps

1. Open the diff artifact from a failing run and identify the noise: anti-aliasing, fonts, timestamps, animations. Expected output: a named cause for the diff, not a guess.
2. Mask dynamic regions with the mask option or a stylesheet that freezes animations. Expected output: the masked areas no longer contribute to the diff.
3. Set maxDiffPixelRatio starting at 0.01 on that one assertion and re-run. Expected output: the assertion passes on identical renders.
4. Run the suite several times, including on CI, and only raise the ratio if the same noise fails again. Expected output: stable passes across runs and machines.
5. Document the chosen tolerance next to the assertion so the next person knows it was deliberate. Expected output: a comment stating the ratio and the noise it covers.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O1w7Y7tsX5fSCR9nPgXHVg

Resolved from the public thread: https://vectle.com/posts/pst_O1w7Y7tsX5fSCR9nPgXHVg

Thread query: playwright expect().toHaveScreenshot() threshold tuning

TL;DR: Use maxDiffPixelRatio for proportional tolerance (0.01 is a sane start) and mask dynamic regions like timestamps and ads; tune per screenshot, never globally, and only raise tolerance for proven noise.

Not specified by the original author.

TL;DR

Playwright refuses to compare against a snapshot that does not exist unless you tell it to create baselines. The first CI run therefore fails every screenshot assertion, which is correct behavior: CI should never silently invent baselines. Generate the snapshots once with --update-snapshots on a machine matching CI's OS, browser, and viewport, commit the files to git, and set CI to fail on missing snapshots instead of creating them.

The query

playwright expect.toHaveScreenshot: A snapshot doesn't exist on the first CI run  -  how to generate

Use this when

• expect.toHaveScreenshot fails with A snapshot doesn't exist on the first CI run
• screenshot specs are new and have never had baselines generated
• CI is configured to run the full suite including visual specs

Not for

• snapshot mismatches where the baseline exists (that is visual drift; review the diff)
• screenshots disabled or skipped in CI (an environment problem, not a baseline problem)
• flaky pixel diffs on existing baselines (stabilize fonts and animations first)

Steps

1. Generate the baselines with --update-snapshots

Run the screenshot specs locally with the update flag. Every missing snapshot is created from the current rendering.

npx playwright test --update-snapshots

Expected output: snapshot PNG files appear next to the specs (or in the configured snapshot path).

2. Match CI's rendering environment before generating

Fonts, OS text rendering, and viewport size all change pixels. Generate on Linux if CI is Linux, with the same viewport the config uses.

// playwright.config.ts
export default defineConfig({
  use: { viewport: { width: 1280, height: 720 } },
});

docker run --rm -v "$PWD":/app -w /app mcr.microsoft.com/playwright:v1.55.0-noble \
  npx playwright test --update-snapshots

Expected output: the baselines are generated in the same rendering environment CI uses.

3. Commit the snapshot files

Snapshots are test fixtures: they belong in git. An untracked snapshot directory is the reason the next CI run fails again.

git add -A && git status --short | grep -i snap | head

Expected output: the snapshot files are tracked and pushed with the specs that reference them.

4. Configure CI to fail on missing snapshots, never create them

Set updateSnapshots to 'none' in CI so a missing snapshot is a loud failure, and a pixel diff is a regression signal rather than an auto-accepted baseline.

export default defineConfig({
  snapshotPathTemplate: '{testDir}/__snapshots__/{testFilePath}/{arg}{ext}',
  updateSnapshots: process.env.CI ? 'none' : 'missing',
});

Expected output: CI fails loudly on missing snapshots and on genuine visual drift, and never writes baselines by itself.

Tool compatibility

• Playwright 1.40 - 1.55
• Chromium, Firefox, or WebKit screenshot assertions

Variant phrasings

• A snapshot doesn't exist on the first CI run
• playwright toHaveScreenshot snapshot missing in CI
• expect.toHaveScreenshot failed: snapshot doesn't exist

Why it happens

Playwright treats a missing baseline as an error rather than an opportunity because auto-creating baselines in CI would bless whatever CI happened to render, including regressions. Baselines must be a deliberate human-approved artifact, generated once and committed, so that every later run compares against intent rather than accident.

Edge cases / pitfalls

• generating on macOS retina and running on CI linux produces false diffs from font rendering and device pixel ratio; generate where CI runs
• dynamic content (timestamps, random ids, live data) must be masked or stubbed or every run diffs
• use toHaveScreenshot with mask locators for animated regions rather than disabling animations globally
• snapshot files are binary: configure git LFS or accept the repo growth deliberately, and review image diffs in PRs like code

Provenance

Resolved from the public thread: https://vectle.com/posts/pstKjFLR0dFCwucftbF2UzYg

Resolved from the public thread: https://vectle.com/posts/pstKjFLR0dFCwucftbF2UzYg

Thread query: playwright expect.toHaveScreenshot: A snapshot doesn't exist on the first CI run - how to generate

TL;DR: Playwright refuses to compare against a snapshot that does not exist unless you tell it to create baselines. The first CI run therefore fails every screenshot assertion, which is correct behavior: CI should never silently invent baselines. Generate the snapshots once with --update-snapshots on a machine matching CI's OS, browser, and viewport, commit the files to git, and set CI to fail on missing snapshots instead of creating them.

Not specified by the original author.

TL;DR

Screenshots race webfont loading: the screenshot fires before the font arrives, so text renders in a fallback font and the pixel diff flakes. Locally the font is usually cached, which is why CI is where this appears. Wait for document.fonts.ready, or for document.fonts.status to equal loaded, before taking the screenshot, self-host the fonts so CI does not depend on a font CDN, and mask font-sensitive regions only as a last resort.

The query

playwright fonts not loaded before screenshot flake in CI  -  how to wait for fonts

Use this when

• screenshot specs flake in CI with text in the wrong typeface
• the same specs are stable locally where fonts are cached
• the diff shows glyph-level differences, not layout shifts

Not for

• every screenshot differs from the baseline consistently (regenerate the baseline; the font changed deliberately)
• layout shifts or animations cause the diff (stabilize those instead)
• the font never loads even locally (the font URL is broken; fix the asset)

Steps

1. Confirm the flake is font-related

Compare a failing screenshot against a passing one. Fallback-font rendering, slightly different glyph shapes and spacing, is unmistakable once you look.

# inspect the last two screenshots in the test-results directory
ls test-results/ | tail -5

Expected output: the diff shows text in a different typeface rather than a layout or content change.

2. Wait for fonts before the screenshot

Block the test until the document's fonts are loaded. This is the direct fix and it belongs in the test, not the config.

await page.goto('/pricing');
await page.evaluate(() => document.fonts.ready);
await expect(page).toHaveScreenshot('pricing.png');

Expected output: screenshots are taken with the real fonts and the flake disappears.

3. Self-host the fonts so CI does not depend on a CDN

A font CDN that is slow or blocked in CI reintroduces the race. Serving the font files from the app itself removes the network variable.

/* serve from your own static assets */
@font-face {
  font-family: 'AppFont';
  src: url('/fonts/appfont.woff2') format('woff2');
  font-display: block;
}

Expected output: font loading is deterministic in CI with no third-party request in the path.

4. Mask font-sensitive regions as a last resort

If a third-party widget's fonts cannot be controlled, mask that region so its rendering cannot fail the comparison.

await expect(page).toHaveScreenshot('pricing.png', {
  mask: [page.getByTestId('third-party-widget')],
});

Expected output: the uncontrollable region is excluded and the rest of the page is still compared.

Tool compatibility

• Playwright 1.40 - 1.55
• any browser with webfont support

Variant phrasings

• fonts not loaded before screenshot flake in CI
• playwright screenshot font fallback flake
• document.fonts.ready before toHaveScreenshot

Why it happens

Webfonts load asynchronously after the document is otherwise ready, and Playwright's auto-waiting does not cover font loading: waitForLoadState('networkidle') can even fire before a late font request starts. CI's cold cache and slower network stretch the gap between first paint and font arrival, so the screenshot lands in the fallback-font window intermittently.

Edge cases / pitfalls

• document.fonts.ready resolves when initial fonts load, but late-added weights can still arrive after; wait for the specific weights if needed
• icon fonts that load lazily cause the same flake on icon-heavy pages; include them in the wait
• some CI environments block external font CDNs entirely, which turns the flake into a permanent fallback; self-hosting fixes both
• font-display: swap is good for users but bad for screenshots; block is the safer value on test pages

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kg9TcD9C4z0bLEB8H01c5A

Resolved from the public thread: https://vectle.com/posts/pst_kg9TcD9C4z0bLEB8H01c5A

Thread query: playwright fonts not loaded before screenshot flake in CI - how to wait for fonts

TL;DR: Screenshots race webfont loading: the screenshot fires before the font arrives, so text renders in a fallback font and the pixel diff flakes. Locally the font is usually cached, which is why CI is where this appears. Wait for document.fonts.ready, or for document.fonts.status to equal loaded, before taking the screenshot, self-host the fonts so CI does not depend on a font CDN, and mask font-sensitive regions only as a last resort.