agent marked the victim test as flaky instead of the polluter test that leaked the state

TL;DR

The test that fails is the victim, not the criminal. When a test passes alone but fails in the suite, the bug is almost always a leaked piece of state from an earlier, passing test. Stop investigating the victim, find the polluter, and move the flake label (or better, delete it) once the polluter's teardown is fixed.

The query

agent marked the victim test as flaky instead of the polluter test that leaked the state

Steps

1. Confirm the victim is innocent

Run the labeled test completely alone, in a fresh process, several times. If it is green every time, the victim's own code is fine and the flake label is wrong.

Expected: the victim passes in isolation, consistently. The label was misdirected.

2. Bisect the tests that run before it

Run the victim after each earlier test, one at a time (or binary-search the list), until it fails. The test that makes it fail is the polluter.

Expected: exactly one earlier test reproduces the victim's failure. That is the polluter.

3. Name the leaked state

Compare the shared world before and after the polluter runs: database contents, files, env vars, open sockets, in-memory caches. Find what the polluter changed and never restored.

Expected: the leaked state described concretely, e.g. "the polluter inserts a user row and its teardown only deletes it when the polluter itself passes."

4. Fix the polluter and un-label the victim

Add unconditional teardown to the polluter (cleanup that runs even on failure). Remove the flake label or quarantine entry from the victim, since it was never flaky.

Expected: the full suite is green, and the victim's record no longer carries a bogus flake history.

Use this when

• The agent labeled or quarantined the test that failed, not a test that ran earlier
• The failing test passes in isolation
• The failure only appears when the suite (or a subset) runs together
• The "flake" has a deterministic trigger: a specific earlier test always causes it

Not for this skill when

• The test fails alone too (then it really is broken, or genuinely flaky, on its own)
• The failure is timing-based and moves around regardless of order (look at timing assertions instead)
• No earlier test reproduces the failure (the polluter may be environmental: check worker config, shared services)
• The agent labeled the right test but proposed the wrong fix (that is a fix-quality problem, not a mislabeling)

Variant phrasings

agent blamed the wrong test because the error surfaced in the consumer

Errors surface where state is read, not where it was corrupted. Always walk upstream from the error.

the flake always involves the same two tests

That is not a flake, it is a polluter/victim pair with a deterministic trigger. Bisect the pair and fix the polluter.

agent quarantined the test that caught the leak

Quarantining the victim is the worst outcome: it deletes the only signal pointing at the polluter. Un-quarantine it and fix the source.

Why it happens

Agents investigate the thing that failed, because the failure is what they can see. The polluter passed, so it looks innocent, and its logs show nothing wrong. The agent's mental model is "failing test = broken test," which is true for unit failures but backwards for state leaks: the broken test is the one that corrupted shared state and got away with it. Labeling the victim flaky then feels evidence-based (it passes on retry in a clean worker), which cements the mistake.

Edge cases

• The polluter only leaks when IT fails: teardown that is skipped on failure is a classic source. Make the polluter's cleanup unconditional and the victim's failures disappear.
• Multiple polluters: no single earlier test reproduces it. Try pairs, or look for a shared fixture they all use.
• The polluter is a fixture, not a test: module- or session-scoped fixtures can leak across tests. Audit fixtures with the same bisect approach.
• The victim was ALSO genuinely flaky: possible but rare. Fix the polluter first; if the victim still flakes alone afterwards, then it has its own problem.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstvREzAH0iXWDRDPcrey7TQ