agent reproduced locally with a newer browser than CI - the bug was a rendering difference in the older version
Fixes unreproducible rendering bugs caused by browser version mismatch between the agent's sandbox and CI. Use when a visual or DOM test fails on CI but passes for the agent and browser versions differ. Covers matching the exact CI browser version, pinning browser versions in CI config, and treating version-specific rendering as real. Not for version-independent failures.
TL;DR
The agent reproduced with a newer browser than CI, so the rendering bug never appeared. Match CI's exact browser version when reproducing (pin it in the CI config and the sandbox), because rendering differences between versions are real bugs for the users on the older version.
The symptom
agent reproduced locally with a newer browser than CI - the bug was a rendering difference in the older versionSteps
1. Compare browser versions
Check the browser version in the CI job logs and in the agent's sandbox. A newer sandbox browser than CI confirms the mismatch. Expected: Two different browser versions documented.
2. Reproduce with CI's exact version
Install or select the CI browser version in the sandbox (version-pinned browser installs, or a container matching CI). Rerun the failing test. Expected: The CI failure reproduces under the older browser.
3. Fix the rendering issue
Address the version-specific rendering difference (CSS, layout, or API behavior that changed between versions), not the test. Expected: The test passes on both the old and new browser versions.
4. Pin browser versions everywhere
Pin the browser version in CI config and make the healer verify its sandbox browser matches before attempting reproduction. Expected: Version mismatches become impossible by construction.
When to use
- Visual or DOM tests fail on CI but pass for the agent
- Browser versions differ between sandbox and CI
- The failure is rendering-shaped (layout, paint, CSS)
When not to use
- Browser versions already match (different cause)
- The failure is logic, not rendering (version matters less)
- CI uses 'latest' intentionally (then the fix is handling version drift, not pinning)
Why it happens
Browsers change rendering behavior between versions: layout engines, default styles, and API support all drift. A newer browser may render correctly what an older one gets wrong. Reproducing on the wrong version tests the wrong browser.
Edge cases
- CI 'latest' tags that float: pin to a digest or version for reproducibility.
- Multiple browsers in the matrix: reproduce with each failing browser version.
- Headless vs headed differences: match that dimension too, not just the version.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_0VzEqh6rngzZp4u2wLN0Ew