agent fixed the failing test in isolation but it only fails when test_zz_auth runs first - the ordering dependency is...

TL;DR The test only fails when testzzauth runs first, so fixing it alone can never work. Run the tests in suite order, find the shared state testzzauth leaves behind, and fix that leak.

agent fixed the failing test in isolation but it only fails when test_zz_auth runs first  -  the ordering dependency is invisible to it

Steps

1. Stop running the test alone. Reproduce in suite order: run the full file (or at least testzzauth followed by the failing test) exactly as CI orders them.

Expected: the failure reproduces only in that order, never solo.

1. Confirm testzzauth is the polluter. Run testzzauth alone (passes), the victim alone (passes), then the two together in order (victim fails). That sequence proves the dependency.

Expected: the pair fails while each solo passes.

1. Find what testzzauth leaks. Diff the process state before and after it: env vars, DB rows, files, global singletons, auth tokens, cached clients.

Expected: you find the specific leftover - e.g. a logged-in session, a mutated global, an uncommitted row - that changes the victim's world.

1. Fix the leak at the source: add teardown to testzzauth, or make the victim independent of the leaked state (fresh fixtures, isolated DB).

Expected: the pair passes in order, and the victim still passes solo.

1. Prevent recurrence: run this file with randomized order in CI so the next ordering dependency fails fast and loudly.

Expected: future polluters get caught immediately instead of hiding for months.

Use this when

• the agent "fixed" a test that only fails in suite order
• the failure mentions state that another test could have set (auth, DB rows, globals)
• the test passes solo and in some orders but not in CI's order
• one specific earlier test always precedes the failure

Not for this skill when

• the test fails standalone too (no ordering involved)
• the dependency is on parallel workers, not execution order
• the ordering is intentional and documented (fixture chains)
• failures are timing-based regardless of order

Variant phrasings

• "test only fails when another test runs first"
• "hidden ordering dependency between tests"
• "agent fixed test in isolation but it fails in suite order"

Why it happens

Test runners execute files in a fixed order and tests share a process. testzzauth logs in, sets globals, or writes rows and never cleans up; the victim test assumes a fresh world. The agent sees the victim's error, debugs the victim, and never looks backward - the polluter is invisible because its own run is green.

Edge cases

• Alphabetical vs file-definition order differs between local and CI; verify which order CI actually uses.
• The polluter may be several tests back, with intermediate tests merely passing the poison along; bisect the prefix, not just the neighbor.
• Auth state is the classic: tokens cached in module scope survive across tests in the same worker.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jzHYlbSyudfzSV-yGWVOHw