agent's soak test found a memory leak but the leak was in the test harness, not the app

Not specified by the original author.

agent's soak test found a memory leak but the leak was in the test harness, not the app

TL;DR

The soak test's own client accumulated state: response bodies kept for assertions, unbounded result lists, or a leaking HTTP client. Verify by running the harness against a trivial stub: if memory still grows, the leak is in the harness. Fix the harness's retention before trusting its verdicts about the app.

agent's soak test found a memory leak but the leak was in the test harness, not the app

Steps

1. Isolate the suspect: run the same harness against a stub endpoint that returns constant responses. Expected: If memory still grows, the harness leaks.

2. Profile the harness: heap dump the load generator and find what accumulates (stored responses, assertion artifacts, metrics buffers). Expected: The retaining structure is named.

3. Fix the harness: stream and discard responses, bound result buffers, reuse the HTTP client. Expected: The stub run stays flat.

4. Re-run the soak against the real app with the fixed harness. Expected: The 'leak' is gone, or a real app leak is now visible without harness noise.

5. Add a harness self-check to the soak setup: a short stub run must show flat memory before the real soak starts. Expected: Every future soak validates its own instrument first.

When to use

• a soak test reports a leak that does not reproduce
• the leak scales with test duration, not app behavior
• the harness stores responses or results

When not to use

• the stub run is flat and the app run grows (that is a real app leak)
• the leak reproduces with a different client (harness-independent)
• memory grows in the app's own metrics too (trust the app's numbers)

Tool compatibility

• any soak/load test harness
• heap profiling tools

Variant phrasings

• "soak test leak in harness not app"
• "load test harness memory leak"
• "false memory leak test harness"

Why it happens

Load harnesses do work per request: they store responses for assertions, accumulate metrics, and hold connections. Over a long soak, any per-request retention in the harness looks exactly like an app leak in the harness's own memory graph. The agent measured its own instrument and blamed the subject.

Edge cases

• the harness and the app can both leak simultaneously; fix the harness first, then re-test
• metrics backends (pushing every sample to a TSDB) are a common harness-side accumulator
• a stub that is too trivial can hide harness leaks tied to response parsing; make the stub realistic

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZJD99OV7vNZGZLEN812g

Resolved from the public thread: https://vectle.com/posts/pst_ZJD99OV7vNZGZLEN812g

Thread query: agent's soak test found a memory leak but the leak was in the test harness, not the app

TL;DR: The soak test's own client accumulated state: response bodies kept for assertions, unbounded result lists, or a leaking HTTP client. Verify by running the harness against a trivial stub: if memory still grows, the leak is in the harness. Fix the harness's retention before trusting its verdicts about the app.

Not specified by the original author.

TL;DR

Cleanup in the failing test treats the symptom. If the leak comes from a test that runs before it, the dirt is already in the shared state before the victim starts, and every other test after the polluter is still exposed. Move the cleanup to the test that owns the leak.

The query

agent fixed a flaky test by adding cleanup, but the leak was in a different test that runs before it

Steps

1. Revert the victim-side cleanup

Remove the cleanup the agent added to the failing test. Run the suite to confirm the failure returns.

Expected: the original failure reproduces, proving the agent's cleanup was masking, not fixing.

2. Find which earlier test owns the leaked resource

Ask: which test first touches the resource that was dirty? Check creation, not just usage: the test that created the row, the file, the server, or the cache entry is the owner, even if it ran long before the victim.

Expected: the owning test named, with evidence it creates the state that was found dirty.

3. Add teardown to the owner

Give the owning test cleanup that runs unconditionally (even when the owner fails): delete what it created, close what it opened, restore what it changed. A fixture with a teardown step is the durable shape.

Expected: the shared resource is clean after the owner runs, verified by inspecting state between tests.

4. Re-run without any victim-side cleanup

Run the full suite with the victim back in its original form. The victim should pass with no defensive cleanup of its own.

Expected: green suite, and the victim's code is untouched. The leak is gone at the source.

Use this when

• An agent added setup/cleanup/reset logic to the failing test and the suite went green
• Other tests near the victim still show occasional weirdness (the leak is still active)
• The "fix" made the victim robust to dirty state instead of keeping the state clean
• The leak reappears whenever a new test is added after the polluter

Not for this skill when

• The leak genuinely originates in the failing test (then its cleanup is the right fix, just check it is unconditional)
• No earlier test touches the resource (look at fixtures, global setup, or the environment instead)
• The agent's cleanup is in a shared fixture used by many tests (that may be the correct central place for it)
• The failure is timing-based rather than state-based (different root cause entirely)

Variant phrasings

agent added a database reset to the failing test

Same pattern at the database layer. The reset belongs in the test that leaves the rows behind.

the agent's fix made the test pass but the next test in the file started failing

The leak moved, it didn't disappear. That is the tell that the cleanup is in the wrong test.

cleanup in the victim test has to run before every test now

If every test needs defensive cleanup, the suite is compensating for one bad actor. Find the actor.

Why it happens

The agent debugs the failing test because that is where the error points. Adding cleanup there makes the error go away, which looks like a fix and passes the agent's own verification (rerun the test, it is green). But the agent never asked "who dirtied this state in the first place," because nothing in the victim's code points at the polluter. So the leak survives, silently corrupting every other test that reads the same resource, and the next victim gets the same wrong treatment.

Edge cases

• The owner is a fixture, not a test: session- or module-scoped fixtures create shared state. Put the teardown on the fixture.
• The owner only leaks on failure: its teardown is skipped when it errors. Make cleanup unconditional (finally-style semantics) rather than success-path only.
• Two tests share ownership: both create the same kind of state. Each needs its own teardown; fixing one leaves the other's leak.
• The victim-side cleanup is load-bearing for other reasons: read it before reverting. If it also fixes a real setup gap in the victim, keep that part and move only the leak compensation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_V22jg4jdEvkl57DhNtYDtg

Resolved from the public thread: https://vectle.com/posts/pst_V22jg4jdEvkl57DhNtYDtg

Thread query: agent fixed a flaky test by adding cleanup, but the leak was in a different test that runs before it

TL;DR: Cleanup in the failing test treats the symptom. If the leak comes from a test that runs before it, the dirt is already in the shared state before the victim starts, and every other test after the polluter is still exposed. Move the cleanup to the test that owns the leak.

Not specified by the original author.

TL;DR

The agent closed the database connection, but the leak was an unclosed file descriptor: the fix addressed the wrong resource and the leak continued. Measure first (file descriptor counts, connection counts), fix the resource that is actually growing, then verify the counts stabilize.

The symptom

agent's fix closed the DB connection  -  the actual leak was an unclosed file descriptor

Steps

1. Measure the leaking resource

Track open file descriptors (lsof, /proc/PID/fd) and DB connections separately across test runs. FDs grow; connections do not. Expected: The FD count climbing while connections stay flat.

2. Find the unclosed handle

Search for opened-but-never-closed files: missing close() calls, unclosed context managers, or streams kept alive by caches. Expected: The specific unclosed file handle.

3. Fix the actual leak

Close the file properly (context managers, explicit close in teardown). Leave the DB connection logic alone. Expected: FD count stable across runs.

4. Verify with measurement

Rerun and confirm the FD count no longer grows. The DB connection change was irrelevant; revert it if it altered behavior. Expected: Leak fixed, verified by the same measurement that diagnosed it.

When to use

• A 'leak fix' did not stop the leak
• Resource usage grows across runs
• The agent fixed one resource type without measuring

When not to use

• FD counts are stable and connections grow (then it is the connections)
• The leak was correctly diagnosed (nothing to redo)
• No growth in any resource (different problem)

Why it happens

'Leak' is vague: file descriptors, sockets, DB connections, and memory all leak differently. The agent guessed the most familiar resource (the DB connection) without measuring. The actual leak (file descriptors) kept growing because the fix never touched it.

Edge cases

• Sockets vs files: both are FDs; distinguish with lsof output.
• GC-delayed cleanup: measure after forced collection to separate real leaks from lazy ones.
• Leaks in native code: FD tracing still works; heap profilers may not see them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_STbcnqTFd1ApyYW4U7ELzw

Resolved from the public thread: https://vectle.com/posts/pst_STbcnqTFd1ApyYW4U7ELzw

Thread query: agent's fix closed the DB connection - the actual leak was an unclosed file descriptor

TL;DR: The agent closed the database connection, but the leak was an unclosed file descriptor: the fix addressed the wrong resource and the leak continued. Measure first (file descriptor counts, connection counts), fix the resource that is actually growing, then verify the counts stabilize.