agent debugged a CI failure using its own sandbox, which had the browsers preinstalled - CI downloads them fresh each...

TL;DR The agent's sandbox had browsers preinstalled, so it never saw the real CI failure: CI downloads browsers fresh every run, and that download step is what breaks. Make the agent replicate CI's browser setup and the failure becomes visible.

agent debugged a CI failure using its own sandbox, which had the browsers preinstalled  -  CI downloads them fresh each run

Steps

1. Read the CI browser-install step. Find the exact install command in the workflow (e.g. the Playwright browser install step) and note the version pin and cache settings.

Expected: you can quote the exact command and whether the browser cache is persisted between runs.

1. Reproduce the failure the way CI sees it. Wipe the local browser cache (or use a clean container), then run the install command exactly as CI does, then run the tests.

Expected: you hit the same failure the agent missed - a download timeout, a version mismatch, or a missing system dependency.

1. Fix the install step, not the tests. Usual fixes: pin the browser version, add a retry around the download, cache the browser dir between runs, or install missing OS libraries in the CI image.

Expected: the install step passes reliably on a cold cache.

1. Teach the agent to distrust its sandbox. Add a runbook note: browser tests must be validated against a fresh browser install, never the sandbox's preinstalled ones.

Expected: the agent's next diagnosis starts from CI's environment, not its own.

1. Rerun the full CI job.

Expected: green from a cold cache, which is the only green that counts.

Use this when

• the agent debugged with preinstalled browsers while CI installs them per run
• CI logs show browser download failures, version mismatches, or missing system libs
• tests pass in the agent sandbox but fail at the browser-setup stage in CI
• the failure is in setup, before any test actually runs

Not for this skill when

• the browser install step passes and tests fail afterward (that is a test bug)
• CI also uses preinstalled browsers (no divergence to fix)
• the failure is a CDN or network outage affecting everyone
• you are debugging the app under test, not the test harness

Variant phrasings

• "playwright browsers not installed in CI but present in local sandbox"
• "CI downloads browsers fresh each run and the download fails"
• "agent could not reproduce browser test failure because of preinstalled browsers"

Why it happens

Agent sandboxes ship with everything preinstalled for speed, including browsers. CI images are usually minimal and install browsers as a job step. Any flakiness in that step - a slow CDN, a version drift between the pinned driver and the downloaded browser, a missing system library in the new image - is invisible to an agent that starts from "browsers already work here".

Edge cases

• Browser caching in CI can hide the problem for weeks, then a cache miss triggers the failure; test with the cache disabled to be sure.
• The downloaded browser version can drift from the driver version if either is unpinned; pin both to the same release.
• Some CI images lack sandbox deps for headless Chrome; the fix is OS packages in the image, not browser flags.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sufXiM0uorB84zEGaC1fEg