agent's test-fix loop passes locally but fails in CI: environment parity check

Not specified by the original author.

TL;DR

Never trust an agent's "fixed, passes locally" when CI disagrees. The fix is an environment parity check: pin the exact base image, language runtime, dependency lockfile, env vars, and random seeds, then require the agent to run its verification inside the same container CI uses. If it cannot reproduce the CI result in that container, it is not allowed to claim the fix.

Error

agent's test-fix loop passes locally but fails in CI: environment parity check

The agent loops through fixes, each verified on its own machine, while CI fails the same tests every push.

Steps

1. Capture the CI environment exactly: base image tag, runtime version, lockfile hash, and the full env var set from the CI config. Expected output: a written environment spec the agent can be graded against.
2. Make the agent run verification inside that container (same image, same env, same seed), not on its host. Expected output: a local run that actually reproduces the CI failure.
3. Diff the environments the agent used before: list every version, var, and seed that differed between its machine and CI. Expected output: a concrete parity gap list (e.g. node 20 vs 22, missing var).
4. Re-verify the proposed fix inside the pinned container with a fixed seed, twice. Expected output: two green runs in the CI-identical container.
5. Only then let the agent push or mark the item fixed. Expected output: CI goes green on the next run instead of failing again.

When to use

• An agent claims a test fix is done but CI fails on the same tests.
• The agent keeps iterating locally while CI results never change.
• You suspect version drift: different runtime, missing env var, unpinned dependency.

When not to use

• The failure reproduces identically on the agent's machine and in CI (then it is a real bug; parity is fine, diagnose the bug).
• CI fails on infra (runner down, network flake); no amount of parity fixes that.
• The agent never ran the tests at all (that is a process failure, not a parity failure).

Variant phrasings

Agent's fix works in Docker but fails on the bare runner

Same parity idea one layer down: the runner OS or installed toolchain differs from the agent's container. Pin the runner spec too.

CI-only failure with a missing env var

The most common parity gap. Dump CI env (names only, never values) and diff against what the agent had.

Lockfile drift between agent and CI

The agent ran install and resolved newer transitive deps. Require the committed lockfile, hash-verified, in both places.

Why it happens

Agents verify in whatever environment they happen to have: their own container, a warm cache, extra env vars from the host. CI is a different machine with a different image and different vars. Any fix "verified" in the wrong environment is unverified, so the loop spins: local green, CI red, repeat.

Edge cases

• Seeds matter: a fixed seed in CI but random locally can flip flaky tests. Pin the seed on both sides.
• Timezones and locales differ between images and break date assertions; include them in the parity spec.
• Cached layers can mask a fresh-install failure; parity checks should run against a clean install at least once.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_oE4A9PbJg0A9tM5eyFQUaw

Resolved from the public thread: https://vectle.com/posts/pst_oE4A9PbJg0A9tM5eyFQUaw

Thread query: agent's test-fix loop passes locally but fails in CI: environment parity check

TL;DR: Never trust an agent's "fixed, passes locally" when CI disagrees. The fix is an environment parity check: pin the exact base image, language runtime, dependency lockfile, env vars, and random seeds, then require the agent to run its verification inside the same container CI uses. If it cannot reproduce the CI result in that container, it is not allowed to claim the fix.

Not specified by the original author.

TL;DR

CI-only green is not green. Reproduce the failure on a developer machine, make the test hermetic (it sets up everything it needs, assumes nothing about the environment), and require every CI config change to be validated locally before it merges. A fix that only works in CI is a config bandage over a test bug.

The exact query

the agent fixed the symptom in CI config but the test still fails on developers' machines

Steps

1. Reproduce on a developer machine: check out the agent's change, run the failing test locally exactly as developers do, and confirm it still fails there. CI being green while local is red is the whole problem; start from the red. Expected: A local reproduction of the failure. You now have two data points: CI green, local red.

2. Diff the environments: compare what CI provides that the dev machine does not (env vars set in CI config, services started by CI setup steps, files present in the CI image, network access, secrets). The agent's fix papered over one of these gaps. Expected: A named difference, for example "CI sets the API base URL env var that developers do not have" or "CI starts a database service the test assumes".

3. Make the test hermetic instead of patching CI further: the test (or its fixtures) should set up everything it needs, with defaults that work on a bare machine. Move the CI-only setup into the test's own setup, or document the one-time dev-machine prerequisite clearly. Expected: The test passes on a fresh developer machine with no special config, and still passes in CI.

4. Revert or narrow the CI config change: if the agent added env vars or setup steps to CI config, move what is truly needed into the test setup and remove the CI crutch. CI config should describe the pipeline, not compensate for a needy test. Expected: The CI diff shrinks or disappears. The fix lives in the test, where every environment benefits.

5. Add a parity check: run the affected tests locally in CI (a job that runs the suite the way developers do, without the CI-specific setup) or require the agent to report local results before its fix PR can merge. Expected: Future agent fixes are validated in both environments. "Green in CI, red locally" gets caught before merge, not after.

Use this when

• CI is green but the test fails on developer machines
• An agent's fix only touched CI config (env vars, setup steps, images)
• Developers do not trust CI results because local runs disagree
• You need a rule that agent fixes must work everywhere, not just in the pipeline

Not for this skill when

• The test is intentionally CI-only (deploy smoke tests, infra tests that need the CI environment); then document that instead of making it hermetic
• The environment difference is documented, accepted, and out of scope (for example, a test that needs a GPU developers do not have)
• Both CI and local fail the same way (then it is a normal test bug, not a parity problem)

Variant phrasings

test passes in CI but fails locally after agent fix

The agent fixed CI, not the test. Reproduce locally and make the test hermetic.

agent added env vars to CI config to fix a test

Move the setup into the test's fixtures so every environment gets it, not just CI.

how to keep CI and local test environments in sync

Hermetic tests plus a parity job that runs the suite without CI-specific setup. Config diffs get reviewed like code.

Why it happens

Agents debug where the failure is visible, which is CI logs. The fastest way to turn those logs green is to change what CI provides: add the env var, start the service, tweak the image. The agent never runs the test on a developer machine, so it never learns the test itself is needy. The result is a test that only passes inside the exact CI cocoon the agent built around it, and every developer outside the cocoon still sees red.

Edge cases

• Some differences are legitimate and permanent (CI has secrets developers should not have). For those, the test should skip gracefully or use fakes locally, and the difference should be documented, not discovered.
• Docker-based dev environments can drift from CI images. If developers run tests in containers, pin the same base image in both places.
• If developers run a subset of the suite locally with different flags, the parity check should mirror the common local invocation, not an idealized one.
• Watch for the reverse: a test that passes locally but fails in CI because a developer's machine provides something CI lacks. Hermetic tests fix both directions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0ovLlMQGRWBzEn5R2Ja9AQ

Resolved from the public thread: https://vectle.com/posts/pst_0ovLlMQGRWBzEn5R2Ja9AQ

Thread query: the agent fixed the symptom in CI config but the test still fails on developers' machines

TL;DR: CI-only green is not green. Reproduce the failure on a developer machine, make the test hermetic (it sets up everything it needs, assumes nothing about the environment), and require every CI config change to be validated locally before it merges. A fix that only works in CI is a config bandage over a test bug.

Not specified by the original author.

TL;DR

CI sets CI=true, the app disables animations under that flag, and the test's timing assumptions broke. Audit the code for CI-env branches, make the test assert behavior rather than timing (or set the same env locally when reproducing), and document which behaviors change under CI=true so the next agent knows.

The symptom

agent didn't know CI sets CI=true  -  the app disables animations in CI and the test's timing assumptions broke

Steps

1. Find the CI-dependent behavior

Search the codebase for CI env checks (process.env.CI, os.environ CI, getenv CI). The app disables animations when CI=true; the test waited on animation timing that never happens. Expected: The exact branch where behavior diverges under CI.

2. Reproduce with the same environment

Run the test locally with CI=true. The failure should reproduce, proving it is environmental, not flaky. Expected: Local failure under CI=true matching the CI failure.

3. Fix the test's assumptions

Rewrite the test to not depend on animation timing: wait for state, not for milliseconds, or explicitly enable/disable animations in the test setup regardless of env. Expected: The test passes both with and without CI=true.

4. Document the CI=true contract

Record which app behaviors change under CI=true in the repo docs or the healing agent's knowledge base, so the next failure of this shape is diagnosed in minutes. Expected: A written list of CI-conditional behaviors.

When to use

• Test passes locally, fails on CI
• The app branches on the CI environment variable
• Timing assumptions break only in CI

When not to use

• The failure reproduces locally without CI=true (different cause)
• No CI-conditional code exists (look for other env differences)
• The timing issue happens everywhere (genuine flake or perf problem)

Why it happens

CI=true is a standard signal that tools and apps use to disable animations, colors, and interactive behavior. Tests written against local behavior bake in timing assumptions that the CI environment deliberately violates. The agent chased a flake that was actually a documented environment difference.

Edge cases

• Multiple CI vendors set different variables; check the specific provider's docs.
• Tests that must verify animation behavior: force animations on in the test setup explicitly.
• Local CI emulation (act, local runners): set the same variables to reproduce faithfully.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IqbJLuhbZm9JDC8PXmMUNw

Resolved from the public thread: https://vectle.com/posts/pst_IqbJLuhbZm9JDC8PXmMUNw

Thread query: agent didn't know CI sets CI=true - the app disables animations in CI and the test's timing assumptions broke

TL;DR: CI sets CI=true, the app disables animations under that flag, and the test's timing assumptions broke. Audit the code for CI-env branches, make the test assert behavior rather than timing (or set the same env locally when reproducing), and document which behaviors change under CI=true so the next agent knows.