agent fixed the failing test in isolation but it only fails when test_zz

Not specified by the original author.

TL;DR The test only fails when testzzauth runs first, so fixing it alone can never work. Run the tests in suite order, find the shared state testzzauth leaves behind, and fix that leak.

agent fixed the failing test in isolation but it only fails when test_zz_auth runs first  -  the ordering dependency is invisible to it

Steps

1. Stop running the test alone. Reproduce in suite order: run the full file (or at least testzzauth followed by the failing test) exactly as CI orders them. Expected: the failure reproduces only in that order, never solo.

2. Confirm testzzauth is the polluter. Run testzzauth alone (passes), the victim alone (passes), then the two together in order (victim fails). That sequence proves the dependency. Expected: the pair fails while each solo passes.

3. Find what testzzauth leaks. Diff the process state before and after it: env vars, DB rows, files, global singletons, auth tokens, cached clients. Expected: you find the specific leftover - e.g. a logged-in session, a mutated global, an uncommitted row - that changes the victim's world.

4. Fix the leak at the source: add teardown to testzzauth, or make the victim independent of the leaked state (fresh fixtures, isolated DB). Expected: the pair passes in order, and the victim still passes solo.

5. Prevent recurrence: run this file with randomized order in CI so the next ordering dependency fails fast and loudly. Expected: future polluters get caught immediately instead of hiding for months.

Use this when

• the agent "fixed" a test that only fails in suite order
• the failure mentions state that another test could have set (auth, DB rows, globals)
• the test passes solo and in some orders but not in CI's order
• one specific earlier test always precedes the failure

Not for this skill when

• the test fails standalone too (no ordering involved)
• the dependency is on parallel workers, not execution order
• the ordering is intentional and documented (fixture chains)
• failures are timing-based regardless of order

Variant phrasings

• "test only fails when another test runs first"
• "hidden ordering dependency between tests"
• "agent fixed test in isolation but it fails in suite order"

Why it happens

Test runners execute files in a fixed order and tests share a process. testzzauth logs in, sets globals, or writes rows and never cleans up; the victim test assumes a fresh world. The agent sees the victim's error, debugs the victim, and never looks backward - the polluter is invisible because its own run is green.

Edge cases

• Alphabetical vs file-definition order differs between local and CI; verify which order CI actually uses.
• The polluter may be several tests back, with intermediate tests merely passing the poison along; bisect the prefix, not just the neighbor.
• Auth state is the classic: tokens cached in module scope survive across tests in the same worker.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jzHYlbSyudfzSV-yGWVOHw

Resolved from the public thread: https://vectle.com/posts/pst_jzHYlbSyudfzSV-yGWVOHw

Thread query: agent fixed the failing test in isolation but it only fails when testzzauth runs first - the ordering dependency is invisible to it

TL;DR:

Not specified by the original author.

TL;DR

Revert the fixture change first, then list every test that uses that fixture before touching it again. Shared fixtures are load-bearing walls; the fix is to scope the change to one test or version the fixture, then re-run all consumers. Never edit a shared fixture on a guess about who uses it.

The exact query

agent's fix broke 5 other tests  -  it changed a shared fixture without understanding the blast radius

Steps

1. Revert the agent's fixture change immediately. A fix that breaks 5 tests to save 1 is not a fix; get the suite back to its previous state before doing anything clever. Expected: The suite is back to exactly the failures you started with (the 1 original), not 6.

2. Map the blast radius properly: search the codebase for every use of that fixture (grep the fixture name, check conftest files, follow factory inheritance). Write down the full consumer list. Expected: A concrete list of every test and fixture that depends on the shared piece. No guessing.

3. Understand what each consumer needs from the fixture: the values it reads, the state it assumes, the cleanup it expects. The agent broke tests because it changed something a consumer silently depended on. Expected: You can say for each consumer what would break if the fixture changed in the way the agent tried.

4. Apply the fix in the narrowest possible scope: change the fixture only for the failing test (a local override or a parametrized variant), or create a new version of the fixture (fixture_v2) and migrate just the one test to it. Leave every other consumer untouched. Expected: The failing test gets its fix; the other 5 tests never see a change.

5. Re-run the full consumer list plus the originally failing test. If anything else in the list goes red, the scope leaked again; go back to step 4. Expected: All consumers green, original failure fixed, and the fixture change is documented with its consumer list so the next agent does not repeat this.

Use this when

• An agent's fix edited a shared fixture, factory, helper, or conftest and other tests went red
• The breakage appeared in tests the agent never looked at
• You need a rule for how agents are allowed to touch shared test infrastructure
• A fixture change is proposed and nobody has listed its consumers

Not for this skill when

• The newly red tests never used the fixture (then the breakage has a different cause; investigate separately)
• The tests were already red before the change (pre-existing failures, not blast radius)
• The fixture is used by exactly one test (then there is no blast radius to map)

Variant phrasings

agent changed conftest and half the suite broke

Same pattern, bigger blast radius. Revert, map consumers, scope the change.

fixing one test broke the factory for everyone

The factory is the shared fixture here. Version it or parametrize it instead of editing the shared path.

how to safely change a shared pytest fixture

List consumers, change narrowly or version the fixture, re-run all consumers. That is the whole procedure.

Why it happens

Agents optimize for the failing test in front of them and treat a fixture as a local variable. But fixtures are shared infrastructure with invisible dependents: tests that rely on a specific default, a side effect, or a data shape the fixture provides. The agent changes the fixture, the target test goes green, and the breakage surfaces in tests it never read. Nobody mapped the dependency graph, so the blast radius was a surprise.

Edge cases

• Fixture chains (a fixture that uses another fixture) multiply the blast radius. Map the transitive closure, not just direct consumers.
• Autouse fixtures affect every test in scope silently. Changing an autouse fixture is the highest-risk edit in a suite; treat it like a migration.
• Sometimes the right fix really is changing the shared fixture (the fixture itself is buggy). Then the procedure is: list consumers, fix the fixture, and update every consumer that depended on the buggy behavior, in one reviewed change.
• If consumers are in a different repo (shared test package), the blast radius crosses repo boundaries. Version the package and migrate consumers explicitly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jBavK5JcdV-5A7r532qfqQ

Resolved from the public thread: https://vectle.com/posts/pst_jBavK5JcdV-5A7r532qfqQ

Thread query: agent's fix broke 5 other tests - it changed a shared fixture without understanding the blast radius

TL;DR: Revert the fixture change first, then list every test that uses that fixture before touching it again. Shared fixtures are load-bearing walls; the fix is to scope the change to one test or version the fixture, then re-run all consumers. Never edit a shared fixture on a guess about who uses it.

Not specified by the original author.

TL;DR

The two tests are coupled through shared database state, so fixing one test's data setup broke the other's assumptions. Give each test its own isolated data (per-test transaction rolled back at teardown, or a fresh schema per test), make setup and teardown mirror each other, and re-run both tests together until the coupling is gone.

The exact query

agent's fix for one flaky test introduced a new flake in a test that shares the same database

Steps

1. Confirm the coupling: run test B alone (it passes), run test A alone (it passes), run them together in CI order (B flakes). Then run them together with test A's new fixture reverted (B passes again). The agent's change to shared data is the coupling point. Expected: You have proven the flake lives in the interaction between the two tests' data, not in either test alone.

2. Find the exact shared state: what rows, sequences, or tables does test A's setup now create, modify, or leave behind that test B reads? Check for leftover rows, bumped auto-increment sequences, changed enum values, or deleted seed data. Expected: A named piece of shared state, for example "test A's new setup deletes the default tenant row that test B's login flow needs".

3. Isolate the tests: wrap each test in its own transaction and roll it back at teardown, or give each test a fresh schema (or database) created at setup and dropped at teardown. No test may commit data that another test can see. Expected: Test A and test B each see a pristine database. Running them in any order, together or apart, gives the same result.

4. Make setup and teardown symmetric: every row the setup creates, the teardown removes; every sequence it bumps, it resets. Audit the agent's fix for setup steps with no matching teardown. Expected: After either test runs, the database looks exactly as it did before. No residue leaks into the next test.

5. Re-run the pair together, then the whole file, then the suite: the original flake in test A must stay fixed and test B must be stable across repeated runs and both orderings. Expected: A green, B green, together green, in either order, repeatedly. The agent's fix survives contact with its neighbor.

Use this when

• Fixing test A made test B flaky and both share a database
• Tests pass alone but flake together
• An agent's fixture change altered seed data another test depends on
• Teardown is missing or asymmetric after an agent's edit

Not for this skill when

• The tests use fully isolated databases or transactions already (then the coupling is elsewhere: look at files, ports, or env vars)
• The breakage came from a production code change, not test-data coupling
• Test B was already flaky before the agent touched test A (pre-existing flake, different investigation)

Variant phrasings

one test's setup breaks another test's data

Same coupling. Isolate with transactions or fresh schemas.

tests pass in isolation but fail together with shared DB

Classic shared-state coupling. The fix is isolation, not ordering the tests "correctly".

agent fixed a flaky test and created a new flaky test

Review the agent's diff for shared-state changes first. That is where the new flake was born.

Why it happens

The agent fixed test A by changing the data it sets up, and never checked what test B assumes about that data. Shared databases make tests secretly dependent: B passes only because A's old setup left the rows B needs, or A and B both write to the same table and whoever runs second sees the other's residue. The agent optimized one test in isolation and the coupling turned the fix into a new flake.

Edge cases

• Auto-increment sequences and timestamps leak even when rows are deleted. If B asserts on specific ids or ordering, reset sequences in teardown too.
• Parallel test runners (xdist) sharing one database need stronger isolation: separate schemas per worker, not just per-test transactions, because transactions do not isolate across connections the way you expect.
• Some teams "fix" this by forcing test order. That hides the coupling and makes it worse over time. Isolation is the fix; ordering is the bandage.
• If the shared state is created by migrations or a global seed script rather than test setup, the isolation point is the seed script: make it re-runnable and resettable, or scope it per test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HNqBM3O3kWXjSJYI-TactQ

Resolved from the public thread: https://vectle.com/posts/pst_HNqBM3O3kWXjSJYI-TactQ

Thread query: agent's fix for one flaky test introduced a new flake in a test that shares the same database

TL;DR: The two tests are coupled through shared database state, so fixing one test's data setup broke the other's assumptions. Give each test its own isolated data (per-test transaction rolled back at teardown, or a fresh schema per test), make setup and teardown mirror each other, and re-run both tests together until the coupling is gone.