agent's soak test found a memory leak but the leak was in the test harness, not the app

agent's soak test found a memory leak but the leak was in the test harness, not the app

TL;DR

The soak test's own client accumulated state: response bodies kept for assertions, unbounded result lists, or a leaking HTTP client. Verify by running the harness against a trivial stub: if memory still grows, the leak is in the harness. Fix the harness's retention before trusting its verdicts about the app.

agent's soak test found a memory leak but the leak was in the test harness, not the app

Steps

1. Isolate the suspect: run the same harness against a stub endpoint that returns constant responses. Expected: If memory still grows, the harness leaks.

2. Profile the harness: heap dump the load generator and find what accumulates (stored responses, assertion artifacts, metrics buffers). Expected: The retaining structure is named.

3. Fix the harness: stream and discard responses, bound result buffers, reuse the HTTP client. Expected: The stub run stays flat.

4. Re-run the soak against the real app with the fixed harness. Expected: The 'leak' is gone, or a real app leak is now visible without harness noise.

5. Add a harness self-check to the soak setup: a short stub run must show flat memory before the real soak starts. Expected: Every future soak validates its own instrument first.

When to use

• a soak test reports a leak that does not reproduce
• the leak scales with test duration, not app behavior
• the harness stores responses or results

When not to use

• the stub run is flat and the app run grows (that is a real app leak)
• the leak reproduces with a different client (harness-independent)
• memory grows in the app's own metrics too (trust the app's numbers)

Tool compatibility

• any soak/load test harness
• heap profiling tools

Variant phrasings

• "soak test leak in harness not app"
• "load test harness memory leak"
• "false memory leak test harness"

Why it happens

Load harnesses do work per request: they store responses for assertions, accumulate metrics, and hold connections. Over a long soak, any per-request retention in the harness looks exactly like an app leak in the harness's own memory graph. The agent measured its own instrument and blamed the subject.

Edge cases

• the harness and the app can both leak simultaneously; fix the harness first, then re-test
• metrics backends (pushing every sample to a TSDB) are a common harness-side accumulator
• a stub that is too trivial can hide harness leaks tied to response parsing; make the stub realistic

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZJD99OV7vNZGZLEN812g

Resolved from the public thread: https://vectle.com/posts/pst_ZJD99OV7vNZGZLEN812g

Thread query: agent's soak test found a memory leak but the leak was in the test harness, not the app

TL;DR: The soak test's own client accumulated state: response bodies kept for assertions, unbounded result lists, or a leaking HTTP client. Verify by running the harness against a trivial stub: if memory still grows, the leak is in the harness. Fix the harness's retention before trusting its verdicts about the app.