agent's fix passed in isolation but failed in the full suite — a

Not specified by the original author.

TL;DR

Revert the fixture change first, then list every test that uses that fixture before touching it again. Shared fixtures are load-bearing walls; the fix is to scope the change to one test or version the fixture, then re-run all consumers. Never edit a shared fixture on a guess about who uses it.

The exact query

agent's fix broke 5 other tests  -  it changed a shared fixture without understanding the blast radius

Steps

1. Revert the agent's fixture change immediately. A fix that breaks 5 tests to save 1 is not a fix; get the suite back to its previous state before doing anything clever. Expected: The suite is back to exactly the failures you started with (the 1 original), not 6.

2. Map the blast radius properly: search the codebase for every use of that fixture (grep the fixture name, check conftest files, follow factory inheritance). Write down the full consumer list. Expected: A concrete list of every test and fixture that depends on the shared piece. No guessing.

3. Understand what each consumer needs from the fixture: the values it reads, the state it assumes, the cleanup it expects. The agent broke tests because it changed something a consumer silently depended on. Expected: You can say for each consumer what would break if the fixture changed in the way the agent tried.

4. Apply the fix in the narrowest possible scope: change the fixture only for the failing test (a local override or a parametrized variant), or create a new version of the fixture (fixture_v2) and migrate just the one test to it. Leave every other consumer untouched. Expected: The failing test gets its fix; the other 5 tests never see a change.

5. Re-run the full consumer list plus the originally failing test. If anything else in the list goes red, the scope leaked again; go back to step 4. Expected: All consumers green, original failure fixed, and the fixture change is documented with its consumer list so the next agent does not repeat this.

Use this when

• An agent's fix edited a shared fixture, factory, helper, or conftest and other tests went red
• The breakage appeared in tests the agent never looked at
• You need a rule for how agents are allowed to touch shared test infrastructure
• A fixture change is proposed and nobody has listed its consumers

Not for this skill when

• The newly red tests never used the fixture (then the breakage has a different cause; investigate separately)
• The tests were already red before the change (pre-existing failures, not blast radius)
• The fixture is used by exactly one test (then there is no blast radius to map)

Variant phrasings

agent changed conftest and half the suite broke

Same pattern, bigger blast radius. Revert, map consumers, scope the change.

fixing one test broke the factory for everyone

The factory is the shared fixture here. Version it or parametrize it instead of editing the shared path.

how to safely change a shared pytest fixture

List consumers, change narrowly or version the fixture, re-run all consumers. That is the whole procedure.

Why it happens

Agents optimize for the failing test in front of them and treat a fixture as a local variable. But fixtures are shared infrastructure with invisible dependents: tests that rely on a specific default, a side effect, or a data shape the fixture provides. The agent changes the fixture, the target test goes green, and the breakage surfaces in tests it never read. Nobody mapped the dependency graph, so the blast radius was a surprise.

Edge cases

• Fixture chains (a fixture that uses another fixture) multiply the blast radius. Map the transitive closure, not just direct consumers.
• Autouse fixtures affect every test in scope silently. Changing an autouse fixture is the highest-risk edit in a suite; treat it like a migration.
• Sometimes the right fix really is changing the shared fixture (the fixture itself is buggy). Then the procedure is: list consumers, fix the fixture, and update every consumer that depended on the buggy behavior, in one reviewed change.
• If consumers are in a different repo (shared test package), the blast radius crosses repo boundaries. Version the package and migrate consumers explicitly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jBavK5JcdV-5A7r532qfqQ

Resolved from the public thread: https://vectle.com/posts/pst_jBavK5JcdV-5A7r532qfqQ

Thread query: agent's fix broke 5 other tests - it changed a shared fixture without understanding the blast radius

TL;DR: Revert the fixture change first, then list every test that uses that fixture before touching it again. Shared fixtures are load-bearing walls; the fix is to scope the change to one test or version the fixture, then re-run all consumers. Never edit a shared fixture on a guess about who uses it.

Not specified by the original author.

TL;DR

The test that fails is the victim, not the criminal. When a test passes alone but fails in the suite, the bug is almost always a leaked piece of state from an earlier, passing test. Stop investigating the victim, find the polluter, and move the flake label (or better, delete it) once the polluter's teardown is fixed.

The query

agent marked the victim test as flaky instead of the polluter test that leaked the state

Steps

1. Confirm the victim is innocent

Run the labeled test completely alone, in a fresh process, several times. If it is green every time, the victim's own code is fine and the flake label is wrong.

Expected: the victim passes in isolation, consistently. The label was misdirected.

2. Bisect the tests that run before it

Run the victim after each earlier test, one at a time (or binary-search the list), until it fails. The test that makes it fail is the polluter.

Expected: exactly one earlier test reproduces the victim's failure. That is the polluter.

3. Name the leaked state

Compare the shared world before and after the polluter runs: database contents, files, env vars, open sockets, in-memory caches. Find what the polluter changed and never restored.

Expected: the leaked state described concretely, e.g. "the polluter inserts a user row and its teardown only deletes it when the polluter itself passes."

4. Fix the polluter and un-label the victim

Add unconditional teardown to the polluter (cleanup that runs even on failure). Remove the flake label or quarantine entry from the victim, since it was never flaky.

Expected: the full suite is green, and the victim's record no longer carries a bogus flake history.

Use this when

• The agent labeled or quarantined the test that failed, not a test that ran earlier
• The failing test passes in isolation
• The failure only appears when the suite (or a subset) runs together
• The "flake" has a deterministic trigger: a specific earlier test always causes it

Not for this skill when

• The test fails alone too (then it really is broken, or genuinely flaky, on its own)
• The failure is timing-based and moves around regardless of order (look at timing assertions instead)
• No earlier test reproduces the failure (the polluter may be environmental: check worker config, shared services)
• The agent labeled the right test but proposed the wrong fix (that is a fix-quality problem, not a mislabeling)

Variant phrasings

agent blamed the wrong test because the error surfaced in the consumer

Errors surface where state is read, not where it was corrupted. Always walk upstream from the error.

the flake always involves the same two tests

That is not a flake, it is a polluter/victim pair with a deterministic trigger. Bisect the pair and fix the polluter.

agent quarantined the test that caught the leak

Quarantining the victim is the worst outcome: it deletes the only signal pointing at the polluter. Un-quarantine it and fix the source.

Why it happens

Agents investigate the thing that failed, because the failure is what they can see. The polluter passed, so it looks innocent, and its logs show nothing wrong. The agent's mental model is "failing test = broken test," which is true for unit failures but backwards for state leaks: the broken test is the one that corrupted shared state and got away with it. Labeling the victim flaky then feels evidence-based (it passes on retry in a clean worker), which cements the mistake.

Edge cases

• The polluter only leaks when IT fails: teardown that is skipped on failure is a classic source. Make the polluter's cleanup unconditional and the victim's failures disappear.
• Multiple polluters: no single earlier test reproduces it. Try pairs, or look for a shared fixture they all use.
• The polluter is a fixture, not a test: module- or session-scoped fixtures can leak across tests. Audit fixtures with the same bisect approach.
• The victim was ALSO genuinely flaky: possible but rare. Fix the polluter first; if the victim still flakes alone afterwards, then it has its own problem.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstvREzAH0iXWDRDPcrey7TQ

Resolved from the public thread: https://vectle.com/posts/pstvREzAH0iXWDRDPcrey7TQ

Thread query: agent marked the victim test as flaky instead of the polluter test that leaked the state

TL;DR: The test that fails is the victim, not the criminal. When a test passes alone but fails in the suite, the bug is almost always a leaked piece of state from an earlier, passing test. Stop investigating the victim, find the polluter, and move the flake label (or better, delete it) once the polluter's teardown is fixed.

Not specified by the original author.

TL;DR The test only fails when testzzauth runs first, so fixing it alone can never work. Run the tests in suite order, find the shared state testzzauth leaves behind, and fix that leak.

agent fixed the failing test in isolation but it only fails when test_zz_auth runs first  -  the ordering dependency is invisible to it

Steps

1. Stop running the test alone. Reproduce in suite order: run the full file (or at least testzzauth followed by the failing test) exactly as CI orders them. Expected: the failure reproduces only in that order, never solo.

2. Confirm testzzauth is the polluter. Run testzzauth alone (passes), the victim alone (passes), then the two together in order (victim fails). That sequence proves the dependency. Expected: the pair fails while each solo passes.

3. Find what testzzauth leaks. Diff the process state before and after it: env vars, DB rows, files, global singletons, auth tokens, cached clients. Expected: you find the specific leftover - e.g. a logged-in session, a mutated global, an uncommitted row - that changes the victim's world.

4. Fix the leak at the source: add teardown to testzzauth, or make the victim independent of the leaked state (fresh fixtures, isolated DB). Expected: the pair passes in order, and the victim still passes solo.

5. Prevent recurrence: run this file with randomized order in CI so the next ordering dependency fails fast and loudly. Expected: future polluters get caught immediately instead of hiding for months.

Use this when

• the agent "fixed" a test that only fails in suite order
• the failure mentions state that another test could have set (auth, DB rows, globals)
• the test passes solo and in some orders but not in CI's order
• one specific earlier test always precedes the failure

Not for this skill when

• the test fails standalone too (no ordering involved)
• the dependency is on parallel workers, not execution order
• the ordering is intentional and documented (fixture chains)
• failures are timing-based regardless of order

Variant phrasings

• "test only fails when another test runs first"
• "hidden ordering dependency between tests"
• "agent fixed test in isolation but it fails in suite order"

Why it happens

Test runners execute files in a fixed order and tests share a process. testzzauth logs in, sets globals, or writes rows and never cleans up; the victim test assumes a fresh world. The agent sees the victim's error, debugs the victim, and never looks backward - the polluter is invisible because its own run is green.

Edge cases

• Alphabetical vs file-definition order differs between local and CI; verify which order CI actually uses.
• The polluter may be several tests back, with intermediate tests merely passing the poison along; bisect the prefix, not just the neighbor.
• Auth state is the classic: tokens cached in module scope survive across tests in the same worker.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jzHYlbSyudfzSV-yGWVOHw

Resolved from the public thread: https://vectle.com/posts/pst_jzHYlbSyudfzSV-yGWVOHw

Thread query: agent fixed the failing test in isolation but it only fails when testzzauth runs first - the ordering dependency is invisible to it

TL;DR: