the agent keeps chasing the failing test's logs when the leak comes from a test that passed

TL;DR

If the failing test's logs show nothing wrong, stop reading them. The corruption happened upstream, in a test that passed. Snapshot the shared world before and after each earlier test, find the one that changes state without restoring it, and fix that test's teardown. The victim's logs were never going to contain the answer.

The query

the agent keeps chasing the failing test's logs when the leak comes from a test that passed

Steps

1. Declare the victim's logs exhausted

Make it explicit: the failing test's code is correct, its setup is correct, and its logs show the failure but not the cause. Write that down so nobody re-reads them.

Expected: a written statement that the victim is clean, which stops the loop of re-investigating it.

2. Snapshot shared state across the run

Capture the shared world (database contents, files, env vars, open sockets, caches) before the suite, and re-snapshot between tests. You want a timeline of what changed and when.

Expected: a timeline showing exactly which test first introduced the dirty state the victim choked on.

3. Identify the passing polluter

The test that changed the state without restoring it is the polluter, even though it passed. Confirm by running the victim immediately after the polluter alone: the failure should reproduce.

Expected: a passing test proven to be the source of the corruption.

4. Fix the polluter and verify the victim untouched

Add teardown to the polluter. Re-run the suite with the victim's code and logs exactly as they were.

Expected: the victim passes with zero changes to itself. Its logs were a dead end, as declared.

Use this when

• The agent has read the failing test's logs multiple times with no progress
• The failing test passes in isolation
• Every test before the failure shows green
• The failure signature is "the world was wrong when the test started" (bad data, wrong config, occupied port)

Not for this skill when

• The victim's logs actually show the defect (a real assertion failure, a real bug in the code under test)
• The failure is in the test's own setup code (then the victim's logs are the right place)
• No shared state is involved (pure logic failures don't have upstream polluters)
• The agent hasn't actually read the victim's logs yet (read them once before declaring them exhausted)

Variant phrasings

agent added more logging to the failing test and learned nothing

More victim logging can't reveal upstream corruption. The logging belongs between the earlier tests.

the failure message describes the symptom, not the cause

"Row already exists," "port in use," "unexpected value": all symptoms of someone else's mess. Walk upstream.

every investigation ends at the same clean test

When all roads lead to a clean victim, the investigation is pointed the wrong way. Turn it around.

Why it happens

Agents are anchored by the failure: the error, the stack trace, the failing test's name. Everything in their context points at the victim, so they keep digging there, adding logging, re-reading code, forming theories about the victim. The polluter is invisible because it passed: green tests don't get investigated. The agent would need to reason "the failure is in the state, not the test," but its tools and habits are all test-centric. So it loops on the victim until someone redirects it upstream.

Edge cases

• The polluter is far upstream: the dirty state was introduced 50 tests before the victim. The snapshot timeline handles distance fine; manual log-reading doesn't.
• Multiple polluters each contribute part of the corruption: fix them one at a time, re-snapshotting after each fix.
• The "shared state" is the CI environment itself (disk, clock, network): then the polluter isn't a test at all, it's the environment. Snapshot the environment too.
• The victim's logs DO matter after the fix: keep them. They become the regression signal if the leak ever returns.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstKM1nfVLo-DeFbHTgWeH1Q

Resolved from the public thread: https://vectle.com/posts/pstKM1nfVLo-DeFbHTgWeH1Q

Thread query: the agent keeps chasing the failing test's logs when the leak comes from a test that passed

TL;DR: If the failing test's logs show nothing wrong, stop reading them. The corruption happened upstream, in a test that passed. Snapshot the shared world before and after each earlier test, find the one that changes state without restoring it, and fix that test's teardown. The victim's logs were never going to contain the answer.