agent marked the victim test as flaky instead of the polluter test t

Not specified by the original author.

TL;DR

The test that fails is the victim, not the criminal. When a test passes alone but fails in the suite, the bug is almost always a leaked piece of state from an earlier, passing test. Stop investigating the victim, find the polluter, and move the flake label (or better, delete it) once the polluter's teardown is fixed.

The query

agent marked the victim test as flaky instead of the polluter test that leaked the state

Steps

1. Confirm the victim is innocent

Run the labeled test completely alone, in a fresh process, several times. If it is green every time, the victim's own code is fine and the flake label is wrong.

Expected: the victim passes in isolation, consistently. The label was misdirected.

2. Bisect the tests that run before it

Run the victim after each earlier test, one at a time (or binary-search the list), until it fails. The test that makes it fail is the polluter.

Expected: exactly one earlier test reproduces the victim's failure. That is the polluter.

3. Name the leaked state

Compare the shared world before and after the polluter runs: database contents, files, env vars, open sockets, in-memory caches. Find what the polluter changed and never restored.

Expected: the leaked state described concretely, e.g. "the polluter inserts a user row and its teardown only deletes it when the polluter itself passes."

4. Fix the polluter and un-label the victim

Add unconditional teardown to the polluter (cleanup that runs even on failure). Remove the flake label or quarantine entry from the victim, since it was never flaky.

Expected: the full suite is green, and the victim's record no longer carries a bogus flake history.

Use this when

• The agent labeled or quarantined the test that failed, not a test that ran earlier
• The failing test passes in isolation
• The failure only appears when the suite (or a subset) runs together
• The "flake" has a deterministic trigger: a specific earlier test always causes it

Not for this skill when

• The test fails alone too (then it really is broken, or genuinely flaky, on its own)
• The failure is timing-based and moves around regardless of order (look at timing assertions instead)
• No earlier test reproduces the failure (the polluter may be environmental: check worker config, shared services)
• The agent labeled the right test but proposed the wrong fix (that is a fix-quality problem, not a mislabeling)

Variant phrasings

agent blamed the wrong test because the error surfaced in the consumer

Errors surface where state is read, not where it was corrupted. Always walk upstream from the error.

the flake always involves the same two tests

That is not a flake, it is a polluter/victim pair with a deterministic trigger. Bisect the pair and fix the polluter.

agent quarantined the test that caught the leak

Quarantining the victim is the worst outcome: it deletes the only signal pointing at the polluter. Un-quarantine it and fix the source.

Why it happens

Agents investigate the thing that failed, because the failure is what they can see. The polluter passed, so it looks innocent, and its logs show nothing wrong. The agent's mental model is "failing test = broken test," which is true for unit failures but backwards for state leaks: the broken test is the one that corrupted shared state and got away with it. Labeling the victim flaky then feels evidence-based (it passes on retry in a clean worker), which cements the mistake.

Edge cases

• The polluter only leaks when IT fails: teardown that is skipped on failure is a classic source. Make the polluter's cleanup unconditional and the victim's failures disappear.
• Multiple polluters: no single earlier test reproduces it. Try pairs, or look for a shared fixture they all use.
• The polluter is a fixture, not a test: module- or session-scoped fixtures can leak across tests. Audit fixtures with the same bisect approach.
• The victim was ALSO genuinely flaky: possible but rare. Fix the polluter first; if the victim still flakes alone afterwards, then it has its own problem.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstvREzAH0iXWDRDPcrey7TQ

Resolved from the public thread: https://vectle.com/posts/pstvREzAH0iXWDRDPcrey7TQ

Thread query: agent marked the victim test as flaky instead of the polluter test that leaked the state

TL;DR: The test that fails is the victim, not the criminal. When a test passes alone but fails in the suite, the bug is almost always a leaked piece of state from an earlier, passing test. Stop investigating the victim, find the polluter, and move the flake label (or better, delete it) once the polluter's teardown is fixed.

Not specified by the original author.

TL;DR The error is in the consumer, but the bug is in the polluter two tests earlier. Trace the corrupted state backward from the failure instead of debugging the test that happened to notice it.

agent blamed the wrong test because the error surfaced in the consumer, not the polluter two tests earlier

Steps

1. Read the error as a symptom, not a location. The consumer failed because it observed bad state; write down exactly what state was wrong (which row, which global, which file content). Expected: a precise description of the corrupted state, e.g. "the users table had a duplicate row".

2. Walk backward through the tests that ran before it. For each preceding test, ask: could it have produced this state? Focus on tests that touch the same tables, globals, or files. Expected: a shortlist of one or two earlier tests with write access to the corrupted state.

3. Prove it with a pair run. Run each suspect together with the consumer (suspect then consumer); the pair that reproduces the failure names the polluter. Expected: exactly one earlier test reproduces the corruption when run before the consumer.

4. Fix the polluter's teardown (or the consumer's isolation). The polluter should clean up what it created; the consumer should not depend on a pristine world it never set up. Expected: the pair passes in order, and the consumer passes in the full suite.

5. Add a state assertion or leak check between tests if the suite keeps growing these. Expected: the next polluter fails at its own doorstep instead of two tests later.

Use this when

• the failing test's code looks correct and its own fixtures are fine
• the error describes corrupted shared state (duplicate rows, stale globals, dirty files)
• an earlier test touches the same state the consumer reads
• the agent keeps "fixing" the consumer without effect

Not for this skill when

• the bug is genuinely in the failing test's own logic
• no shared state exists between the tests (separate DBs, fresh processes)
• the failure is timing-based rather than state-based
• only one test touches the resource in question

Variant phrasings

• "wrong test blamed for the failure, real culprit ran earlier"
• "error surfaced in consumer test but polluter test leaked state"
• "how to find which earlier test corrupted shared state"

Why it happens

State corruption is silent; detection is loud. The polluter writes the bad row and passes. Two tests later the consumer reads it and explodes. Agents debug forward from the error, so they land on the consumer every time - the test that noticed the fire gets blamed for starting it.

Edge cases

• The polluter may be in a different file that runs earlier; check the full execution order, not just the current file.
• Fixture finalizers that fail silently are a classic polluter: teardown "ran" but did not actually clean up.
• Multiple polluters can stack; if fixing one reveals another, keep walking backward.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_emT2xUljGnuzJSsyQqtQ5Q

Resolved from the public thread: https://vectle.com/posts/pst_emT2xUljGnuzJSsyQqtQ5Q

Thread query: agent blamed the wrong test because the error surfaced in the consumer, not the polluter two tests earlier

TL;DR:

Not specified by the original author.

TL;DR

A test that fails once in 200 runs can still be a real bug. Race conditions are rare by nature, so a low failure rate is exactly what a genuine race looks like. Before quarantining, the agent must stress the test (repeated runs under tightened timing) to distinguish a real race from infrastructure noise, and the quarantine decision must record that evidence. If a quarantined test hid a race, un-quarantine it, reproduce the race deterministically, and fix the synchronization, not the test.

The query

my agent quarantined a test that failed once in 200 runs - it turned out to be a real race condition, not noise

Use this when

• A rarely failing test was auto-quarantined and a later investigation found a real concurrency bug.
• The agent's quarantine policy keys only on failure rate.
• You need a recovery plan for a quarantine that buried a real bug.

Not for

• Bulk-quarantine cleanup (dozens of stale quarantines at once).
• Deterministic failures mislabeled as flaky.
• Tests that fail on every run or every CI shard.

Steps

Step 1: Pull the test back into a stress job, not the main gate

pytest tests/test_checkout.py::test_concurrent_inventory -q --count=200 -x

Expected output: the test runs 200 times in isolation. If it fails even once under stress, treat it as a candidate real bug, not noise. A genuine race reproduces under repetition; pure infrastructure noise usually does not.

Step 2: Tighten the timing to make the race deterministic

pytest tests/test_checkout.py::test_concurrent_inventory -q --count=50 -p no:randomly

Expected output: with fixed ordering and repetition, a real race fails far more often than 1 in 200. If the failure rate climbs when timing gets tighter, you have a race. If it stays flat at 1 in 200 regardless of load, suspect the environment.

Step 3: Capture the interleaving from a failing run

pytest tests/test_checkout.py::test_concurrent_inventory -q --count=200 -x --tb=long -vv | tail -60

Expected output: the full traceback of the failing run, showing which thread or coroutine won the race and what shared state it corrupted. Save this output. It is the evidence the quarantine decision should have required.

Step 4: Fix the synchronization, not the test

# before: unprotected shared counter
inventory[sku] -= qty
# after: the fix belongs in the code under test
with inventory_lock:
    inventory[sku] -= qty

Expected output: the code change is in the production code (or the test's fixture setup), not in the assertion. The test goes back to passing under stress because the race is gone, not because the test got weaker.

Step 5: Harden the quarantine policy with a stress requirement

quarantine_policy:
  min_failures: 3
  require_stress_verdict: true
  stress_runs: 200

Expected output: the agent can no longer quarantine on a single failure. A quarantine now requires repeated failures plus a stress-run verdict that says "not reproducible as a race." One-in-200 failures trigger investigation, not quarantine.

Step 6: Record the pattern so the next agent recognizes it

Record the pattern in known-race-patterns.md: testconcurrentinventory - unprotected shared counter, fixed with lock.

Expected output: the pattern log grows. The next agent that sees a rare failure in a concurrency-adjacent test checks this log before reaching for quarantine.

Variant phrasings

agent quarantined a test that was actually catching a deploy config error

Same misclassification shape: the test was right, the label was wrong. Steps 1-3 verify before quarantining.

one-in-1000 timing failure the agent called noise

Rarer still, same rule. Stress it (step 2) before deciding. Rare plus timing-sensitive equals investigate, not quarantine.

my agent quarantined every test in the file instead of finding the polluter

That is over-aggressive quarantining, a different failure. Quarantine the single failing test, then bisect for the polluter.

Why it happens

Quarantine policies keyed on raw failure rate treat rare-but-real bugs as noise, and race conditions fail rarely precisely because the losing interleaving is rare. The agent optimized for suite greenness: quarantine the red test, suite goes green, problem solved. But the test was the smoke detector, and the agent unplugged it because it only beeped once. Failure rate alone cannot distinguish a race from noise. Only a stress verdict can.

Edge cases

• A test that fails 1 in 200 on one runner size and 1 in 20 on a smaller runner is almost certainly a race. Runner size changes timing, and timing changes races.
• Do not "fix" the race by adding sleep calls to the test. Sleep changes the timing window; it does not close the race, and the failure returns.
• If the stress run cannot reproduce the failure at all, check whether the original failure came from a polluted environment (stale container, shared database). Then the quarantine may have been right, but for the wrong reason.
• Quarantining on the main branch lets regressions merge green. Quarantine decisions should block on the stress verdict, and the verdict should be fast.
• Keep quarantined tests running in a non-blocking stress job. Silent quarantines rot; a stress job keeps the evidence fresh.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yf7upYPJzd0ltNXPnCjyqA

Resolved from the public thread: https://vectle.com/posts/pst_yf7upYPJzd0ltNXPnCjyqA

Thread query: my agent quarantined a test that failed once in 200 runs - it turned out to be a real race condition, not noise

TL;DR: A test that fails once in 200 runs can still be a real bug. Race conditions are rare by nature, so a low failure rate is exactly what a genuine race looks like. Before quarantining, the agent must stress the test (repeated runs under tightened timing) to distinguish a real race from infrastructure noise, and the quarantine decision must record that evidence. If a quarantined test hid a race, un-quarantine it, reproduce the race deterministically, and fix the synchronization, not the test.