## TL;DR
CI-only green is not green. Reproduce the failure on a developer machine, make the test hermetic (it sets up everything it needs, assumes nothing about the environment), and require every CI config change to be validated locally before it merges. A fix that only works in CI is a config bandage over a test bug.

## The exact query
```text
the agent fixed the symptom in CI config but the test still fails on developers' machines
```

## Steps
1. Reproduce on a developer machine: check out the agent's change, run the failing test locally exactly as developers do, and confirm it still fails there. CI being green while local is red is the whole problem; start from the red.
   Expected: A local reproduction of the failure. You now have two data points: CI green, local red.
2. Diff the environments: compare what CI provides that the dev machine does not (env vars set in CI config, services started by CI setup steps, files present in the CI image, network access, secrets). The agent's fix papered over one of these gaps.
   Expected: A named difference, for example "CI sets the API base URL env var that developers do not have" or "CI starts a database service the test assumes".
3. Make the test hermetic instead of patching CI further: the test (or its fixtures) should set up everything it needs, with defaults that work on a bare machine. Move the CI-only setup into the test's own setup, or document the one-time dev-machine prerequisite clearly.
   Expected: The test passes on a fresh developer machine with no special config, and still passes in CI.
4. Revert or narrow the CI config change: if the agent added env vars or setup steps to CI config, move what is truly needed into the test setup and remove the CI crutch. CI config should describe the pipeline, not compensate for a needy test.
   Expected: The CI diff shrinks or disappears. The fix lives in the test, where every environment benefits.
5. Add a parity check: run the affected tests locally in CI (a job that runs the suite the way developers do, without the CI-specific setup) or require the agent to report local results before its fix PR can merge.
   Expected: Future agent fixes are validated in both environments. "Green in CI, red locally" gets caught before merge, not after.

## Use this when
- CI is green but the test fails on developer machines
- An agent's fix only touched CI config (env vars, setup steps, images)
- Developers do not trust CI results because local runs disagree
- You need a rule that agent fixes must work everywhere, not just in the pipeline

## Not for this skill when
- The test is intentionally CI-only (deploy smoke tests, infra tests that need the CI environment); then document that instead of making it hermetic
- The environment difference is documented, accepted, and out of scope (for example, a test that needs a GPU developers do not have)
- Both CI and local fail the same way (then it is a normal test bug, not a parity problem)

## Variant phrasings
### test passes in CI but fails locally after agent fix
The agent fixed CI, not the test. Reproduce locally and make the test hermetic.

### agent added env vars to CI config to fix a test
Move the setup into the test's fixtures so every environment gets it, not just CI.

### how to keep CI and local test environments in sync
Hermetic tests plus a parity job that runs the suite without CI-specific setup. Config diffs get reviewed like code.

## Why it happens
Agents debug where the failure is visible, which is CI logs. The fastest way to turn those logs green is to change what CI provides: add the env var, start the service, tweak the image. The agent never runs the test on a developer machine, so it never learns the test itself is needy. The result is a test that only passes inside the exact CI cocoon the agent built around it, and every developer outside the cocoon still sees red.

## Edge cases
- Some differences are legitimate and permanent (CI has secrets developers should not have). For those, the test should skip gracefully or use fakes locally, and the difference should be documented, not discovered.
- Docker-based dev environments can drift from CI images. If developers run tests in containers, pin the same base image in both places.
- If developers run a subset of the suite locally with different flags, the parity check should mirror the common local invocation, not an idealized one.
- Watch for the reverse: a test that passes locally but fails in CI because a developer's machine provides something CI lacks. Hermetic tests fix both directions.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0ovLlMQGRWBzEn5R2Ja9AQ