playwright test kept timing out in CI but passed locally - the agent couldn't reproduce because it didn't know CI...

TL;DR

Your laptop has 8 CPUs; CI has 1. That is the whole bug most of the time. Reproduce with constrained CPU (run Playwright with a single worker under CPU throttling), read the trace to see what was actually slow, and set timeouts and parallelism for the CI hardware you have, not the laptop you wish you had.

The exact query

playwright test kept timing out in CI but passed locally  -  the agent couldn't reproduce because it didn't know CI runs with 1 CPU

Steps

1. Learn the CI hardware: find the runner's CPU count and memory in the CI config or the job's system info. Write it down next to your laptop's specs. The gap between them is your prime suspect.

Expected: Concrete numbers, for example "CI: 1 CPU, 2GB RAM. Laptop: 8 CPU, 32GB RAM." The agent's blind spot is now visible.

1. Reproduce under constraint: run the Playwright test locally with --workers=1 and CPU throttling (browser DevTools throttling, or a container with --cpus=1). This simulates the CI timing profile instead of your laptop's.

Expected: The test now times out locally too, or at least runs dramatically slower. You have reproduced the CI failure.

1. Read the trace, not the timeout message: open the Playwright trace from the CI run and find which step actually consumed the time (slow navigation, an animation that never settles under throttled CPU, a waitForSelector on an element blocked behind a busy main thread).

Expected: A named slow step with its duration, for example "homepage load took 24s of the 30s timeout because the throttled main thread could not hydrate in time".

1. Fix for the constrained hardware: reduce parallelism for that spec, split the spec file, raise that test's timeout based on measured throttled timing (not a guess), or lighten what the page does under test (smaller fixtures, fewer animations). Do not just bump the global timeout.

Expected: The test passes under throttled conditions with margin to spare, and the suite's total time is accounted for honestly.

1. Teach the agent the hardware facts: put the CI runner spec (CPU, RAM) in the repo's testing docs or CI config comments, and require the agent to read it before attempting any CI-only reproduction. A reproduction that ignores the hardware is not a reproduction.

Expected: The next CI-only Playwright timeout starts with "CI has 1 CPU" instead of three wasted local reruns.

Use this when

• Playwright tests time out in CI but pass locally
• CI runners have far fewer CPUs than developer machines
• An agent said "cannot reproduce" for a CI-only timeout
• You are setting Playwright workers or timeouts for CI

Not for this skill when

• The timeout reproduces on a fast machine too (then it is real app slowness or a network issue, not a CPU constraint)
• The failure is a hard error (selector never matches, page crashes) rather than a timeout
• CI and local have equivalent hardware (then look at network, viewport, or browser version differences)

Variant phrasings

playwright timeout only on CI with limited CPU

Constrain locally, read the trace, size for the CI hardware. Same procedure.

agent can't reproduce CI-only playwright failure

Ask what hardware the agent reproduced on. If the answer is not "the CI spec", the reproduction was invalid.

how many playwright workers for 1 CPU CI runner

Start with --workers=1 and measure. More workers on 1 CPU just timeshare the same core and make timing worse.

Why it happens

Playwright tests are timing-sensitive by nature: animations, hydration, network, and rendering all compete for the main thread. On a 1-CPU runner everything runs slower and more serially, so operations that take 2 seconds on a laptop take 15 in CI. The agent reproduces on unconstrained hardware, sees everything pass, and concludes the failure is mysterious. It was never mysterious; the agent just never ran the test the way CI runs it.

Edge cases

• CPU throttling in DevTools slows the renderer but not perfectly; a container with a real CPU limit (--cpus=1) is a more faithful reproduction.
• Some CI providers burst CPU briefly then throttle. If the test passes in the first minute of the CI run and fails later, you may be seeing burst-then-throttle, not a steady 1 CPU.
• Raising the timeout without reducing work just moves the failure to the job-level timeout. Budget the spec's total time, not just the single timeout.
• If the page under test is heavy by design, consider a dedicated larger runner for the E2E job rather than torturing the test to fit 1 CPU. Hardware is sometimes the right fix.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstH8dQlXQojnSF2Uk5sibQg