agent's test-fix loop passes locally but fails in CI: environment parity check

TL;DR

Never trust an agent's "fixed, passes locally" when CI disagrees. The fix is an environment parity check: pin the exact base image, language runtime, dependency lockfile, env vars, and random seeds, then require the agent to run its verification inside the same container CI uses. If it cannot reproduce the CI result in that container, it is not allowed to claim the fix.

Error

agent's test-fix loop passes locally but fails in CI: environment parity check

The agent loops through fixes, each verified on its own machine, while CI fails the same tests every push.

Steps

1. Capture the CI environment exactly: base image tag, runtime version, lockfile hash, and the full env var set from the CI config. Expected output: a written environment spec the agent can be graded against.
2. Make the agent run verification inside that container (same image, same env, same seed), not on its host. Expected output: a local run that actually reproduces the CI failure.
3. Diff the environments the agent used before: list every version, var, and seed that differed between its machine and CI. Expected output: a concrete parity gap list (e.g. node 20 vs 22, missing var).
4. Re-verify the proposed fix inside the pinned container with a fixed seed, twice. Expected output: two green runs in the CI-identical container.
5. Only then let the agent push or mark the item fixed. Expected output: CI goes green on the next run instead of failing again.

When to use

• An agent claims a test fix is done but CI fails on the same tests.
• The agent keeps iterating locally while CI results never change.
• You suspect version drift: different runtime, missing env var, unpinned dependency.

When not to use

• The failure reproduces identically on the agent's machine and in CI (then it is a real bug; parity is fine, diagnose the bug).
• CI fails on infra (runner down, network flake); no amount of parity fixes that.
• The agent never ran the tests at all (that is a process failure, not a parity failure).

Variant phrasings

Agent's fix works in Docker but fails on the bare runner

Same parity idea one layer down: the runner OS or installed toolchain differs from the agent's container. Pin the runner spec too.

CI-only failure with a missing env var

The most common parity gap. Dump CI env (names only, never values) and diff against what the agent had.

Lockfile drift between agent and CI

The agent ran install and resolved newer transitive deps. Require the committed lockfile, hash-verified, in both places.

Why it happens

Agents verify in whatever environment they happen to have: their own container, a warm cache, extra env vars from the host. CI is a different machine with a different image and different vars. Any fix "verified" in the wrong environment is unverified, so the loop spins: local green, CI red, repeat.

Edge cases

• Seeds matter: a fixed seed in CI but random locally can flip flaky tests. Pin the seed on both sides.
• Timezones and locales differ between images and break date assertions; include them in the parity spec.
• Cached layers can mask a fresh-install failure; parity checks should run against a clean install at least once.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_oE4A9PbJg0A9tM5eyFQUaw