flaky test passes locally but fails in CI: debugging

TL;DR

CI-only flakes come from slower, smaller, or different environments. Reproduce CI conditions locally (same container, fewer CPUs, headless), then fix the timing or resource assumption.

Error

(Not an error; a debugging workflow. Symptom: green locally, red sometimes in CI.)

Steps

1. Run the test 20 times locally: for i in $(seq 20); do npx playwright test file.spec.ts; done. Expected: if it never flakes locally, it is environmental.
2. Run in the CI container image locally. Expected: reproduces container-specific flakes.
3. Throttle CPU: run with fewer workers or CPU limits to simulate slow CI. Expected: timing flakes appear.
4. Check the usual suspects: timezone, locale, fonts, screen size, seeded data. Expected: one differs.
5. Fix the assumption (explicit waits, pinned TZ, hermetic data) and verify with 20 CI runs. Expected: the flake is gone where it lived.

When to use

• Flaky only in CI, never locally.
• Intermittent, not consistent.

When not to use

• Consistent CI failure (environment difference, not flake).
• Flaky everywhere (test bug; debug locally).

Tool compatibility

• Any framework; docker for CI reproduction.

Variant phrasings

Test flaky in CI only

The classic report; environment diffing is the method.

Works on my machine test failure

The joke with a real debugging procedure behind it.

Why it happens

CI runners are slower, smaller, and differently configured than laptops. Tests with implicit timing or resource assumptions break there first.

Edge cases

• --repeat-each in Playwright automates the 20-run loop.
• Record CI timings; flakes cluster on slow runs.
• Do not "fix" by raising timeouts globally; fix the assumption.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst3UcqS0vlq2ppkMBsZkOsw