agent fixed a flaky test by adding cleanup, but the leak was in a different test that runs before it

TL;DR

Cleanup in the failing test treats the symptom. If the leak comes from a test that runs before it, the dirt is already in the shared state before the victim starts, and every other test after the polluter is still exposed. Move the cleanup to the test that owns the leak.

The query

agent fixed a flaky test by adding cleanup, but the leak was in a different test that runs before it

Steps

1. Revert the victim-side cleanup

Remove the cleanup the agent added to the failing test. Run the suite to confirm the failure returns.

Expected: the original failure reproduces, proving the agent's cleanup was masking, not fixing.

2. Find which earlier test owns the leaked resource

Ask: which test first touches the resource that was dirty? Check creation, not just usage: the test that created the row, the file, the server, or the cache entry is the owner, even if it ran long before the victim.

Expected: the owning test named, with evidence it creates the state that was found dirty.

3. Add teardown to the owner

Give the owning test cleanup that runs unconditionally (even when the owner fails): delete what it created, close what it opened, restore what it changed. A fixture with a teardown step is the durable shape.

Expected: the shared resource is clean after the owner runs, verified by inspecting state between tests.

4. Re-run without any victim-side cleanup

Run the full suite with the victim back in its original form. The victim should pass with no defensive cleanup of its own.

Expected: green suite, and the victim's code is untouched. The leak is gone at the source.

Use this when

• An agent added setup/cleanup/reset logic to the failing test and the suite went green
• Other tests near the victim still show occasional weirdness (the leak is still active)
• The "fix" made the victim robust to dirty state instead of keeping the state clean
• The leak reappears whenever a new test is added after the polluter

Not for this skill when

• The leak genuinely originates in the failing test (then its cleanup is the right fix, just check it is unconditional)
• No earlier test touches the resource (look at fixtures, global setup, or the environment instead)
• The agent's cleanup is in a shared fixture used by many tests (that may be the correct central place for it)
• The failure is timing-based rather than state-based (different root cause entirely)

Variant phrasings

agent added a database reset to the failing test

Same pattern at the database layer. The reset belongs in the test that leaves the rows behind.

the agent's fix made the test pass but the next test in the file started failing

The leak moved, it didn't disappear. That is the tell that the cleanup is in the wrong test.

cleanup in the victim test has to run before every test now

If every test needs defensive cleanup, the suite is compensating for one bad actor. Find the actor.

Why it happens

The agent debugs the failing test because that is where the error points. Adding cleanup there makes the error go away, which looks like a fix and passes the agent's own verification (rerun the test, it is green). But the agent never asked "who dirtied this state in the first place," because nothing in the victim's code points at the polluter. So the leak survives, silently corrupting every other test that reads the same resource, and the next victim gets the same wrong treatment.

Edge cases

• The owner is a fixture, not a test: session- or module-scoped fixtures create shared state. Put the teardown on the fixture.
• The owner only leaks on failure: its teardown is skipped when it errors. Make cleanup unconditional (finally-style semantics) rather than success-path only.
• Two tests share ownership: both create the same kind of state. Each needs its own teardown; fixing one leaves the other's leak.
• The victim-side cleanup is load-bearing for other reasons: read it before reverting. If it also fixes a real setup gap in the victim, keep that part and move only the leak compensation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_V22jg4jdEvkl57DhNtYDtg