agent blamed the biggest heap object for the leak - it was the cache working as designed

Not specified by the original author.

agent blamed the biggest heap object for the leak - it was the cache working as designed

TL;DR

The biggest object in a heap dump is usually the cache doing its job, not the leak. A leak is growth over time, not size at a moment. Compare dumps across time: the leaking structure grows unboundedly, while the cache plateaus at its configured bound. Blame the growth, not the size.

agent blamed the biggest heap object for the leak - it was the cache working as designed

Steps

1. Take two heap dumps hours apart under similar load. Expected: You have a before and after.

2. Diff the dominators: which structures grew between dumps, and which stayed flat. Expected: The cache is flat at its bound; something else grew.

3. Check the cache's configured bound: max size, TTL, eviction policy. Expected: The cache is behaving as configured.

4. Find the real grower: the structure whose retained size increases without bound across dumps. Expected: The leak is named.

5. Fix the grower (unbounded list, missing eviction, listener accumulation) and leave the cache alone. Expected: Heap stabilizes; the cache keeps doing its job.

When to use

• a leak report names the biggest heap object
• the accused object is a cache
• you have a single heap dump and a conclusion

When not to use

• the cache has no bound configured (then it might genuinely be the leak)
• the biggest object grows without bound across dumps (then it is the leak)
• there is no cache in the app (look elsewhere)

Tool compatibility

• any heap analyzer
• two-dump comparison workflow

Variant phrasings

• "blamed biggest heap object leak"
• "biggest object was cache not leak"
• "heap dump biggest object cache"

Why it happens

Heap dumps show size, and the largest structure draws the eye. But caches are supposed to be large: they are bounded memory doing useful work. A leak is defined by growth, which one dump cannot show. The agent confused 'big' with 'leaking' because it had one snapshot instead of a series.

Edge cases

• a cache with a broken eviction policy is both big and leaking; check the bound is actually enforced
• off-heap caches do not appear in heap dumps at all; the biggest on-heap object may be a red herring twice over
• dominators lie when the leak is many small objects; use class histograms too

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lfbfzunZaiEmg2abAfV4VA

Resolved from the public thread: https://vectle.com/posts/pst_lfbfzunZaiEmg2abAfV4VA

Thread query: agent blamed the biggest heap object for the leak - it was the cache working as designed

TL;DR: The biggest object in a heap dump is usually the cache doing its job, not the leak. A leak is growth over time, not size at a moment. Compare dumps across time: the leaking structure grows unboundedly, while the cache plateaus at its configured bound. Blame the growth, not the size.

Not specified by the original author.

TL;DR

The symptom disappears after each deploy because the agent purges the cache, not because the deploy fixed anything. When the cache warms, the bug returns. The purge is masking a stale-cache or cache-key bug. Stop purging, reproduce with a warm cache, and fix the actual caching logic.

The query

agent loop detected: purging the cache API after every deploy, masking the real bug

Use this when

• Deploys 'fix' the issue for minutes, then it returns.
• The agent purges cache as a standard deploy step.
• The bug only manifests with warm cache.

Not for

• Legitimate invalidation when content actually changed (correct use).
• One-time purges during incidents (deliberate).
• Cache-key bugs with no purge involved (fix the keys directly).

Steps

Step 1: Stop the purge and reproduce warm

Deploy without purging, warm the cache with the failing request pattern, and observe.

Expected result: the bug reproduces reliably with warm cache. You now have a stable repro.

Step 2: Inspect what the cache is actually storing

Check cache keys, vary headers, and TTLs for the affected paths.

Expected result: you find the real defect: wrong cache key collapsing distinct responses, missing vary on auth headers, or a TTL far longer than the content's freshness.

Step 3: Fix the caching logic

Correct the keys, add the missing vary, or shorten the TTL. Purge once, deliberately, to clear the poisoned entries.

Expected result: warm cache serves correct content. The bug is gone with the cache warm, which is the real test.

Step 4: Remove the purge from the deploy ritual

Expected result: deploys no longer purge. If content deploys need invalidation, make it targeted (exact keys) and explicit, not a blanket purge.

Step 5: Add a warm-cache check to the deploy verification

Expected result: post-deploy checks run against warm cache, so a masking purge can never again fake a green deploy.

Why it happens

Purge-then-verify is a comforting ritual: everything looks fresh and correct. But it tests the cold path only. The agent learned that purge makes the red go green and kept doing it, while the warm-path bug compounded. The loop is between the agent's verification method and the actual defect.

Edge cases

• Some deploys genuinely require invalidation (renamed assets, changed APIs). Distinguish 'content changed' from 'bug hunt'.
• The Cache API purge and CDN cache purge are different operations. Know which one the agent was calling.
• Stale-while-revalidate patterns can also mask bugs; check those too.
• Log every purge with a reason. Unexplained purges in the log are the early warning for this loop.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wLrFzbP2f2nEgfXElFICXg

Resolved from the public thread: https://vectle.com/posts/pst_wLrFzbP2f2nEgfXElFICXg

Thread query: agent loop detected: purging the cache API after every deploy, masking the real bug

TL;DR: The symptom disappears after each deploy because the agent purges the cache, not because the deploy fixed anything. When the cache warms, the bug returns. The purge is masking a stale-cache or cache-key bug. Stop purging, reproduce with a warm cache, and fix the actual caching logic.

Not specified by the original author.

agent's memory leak alert fired on JVM heap growth that was just the heap filling to -Xmx

TL;DR

A JVM grows its heap toward -Xmx as needed; that is normal, not a leak. A leak is growth after full GCs that never reclaims, or old-gen growth that never plateaus. Alert on post-GC old-gen trends, not on heap size. The alert fired on physics, not a bug.

agent's memory leak alert fired on JVM heap growth that was just the heap filling to -Xmx

Steps

1. Check the GC logs: after a full GC, does old-gen usage return to a stable level? Expected: If yes, there is no leak; the heap is just filling its allowed space.

2. Plot old-gen after full GCs over days, not heap committed. Expected: A real leak shows a rising floor; normal fill shows a flat floor under a rising ceiling.

3. Retune the alert: fire on the post-GC old-gen trend crossing a threshold, with a minimum observation window. Expected: The alert stops firing on healthy growth.

4. Size -Xmx deliberately: 3-4x the steady-state live set gives headroom without hiding leaks for months. Expected: The heap has room to breathe and leaks still surface.

5. Document the expected sawtooth so the next on-call does not page on it. Expected: The pattern is recognized, not alarming.

When to use

• a leak alert fires on a healthy JVM
• heap grows toward -Xmx and stabilizes
• post-GC old-gen is flat

When not to use

• post-GC old-gen climbs steadily (that is a real leak)
• the JVM actually OOMs (investigate, do not just retune the alert)
• using a collector where old-gen is not the right signal (ZGC/Shenandoah need different metrics)

Tool compatibility

• HotSpot JVM
• GC logging

Variant phrasings

• "JVM heap growth to Xmx not leak"
• "false memory leak alert JVM"
• "heap filling to max normal"

Why it happens

The JVM expands the heap lazily up to -Xmx as the live set needs it, and GC ergonomics prefer growing over collecting aggressively. Watching committed heap size therefore shows growth on every healthy JVM after a restart. The leak signal lives in what survives full collections, not in the ceiling.

Edge cases

• a leak with a huge -Xmx takes longer to OOM; the alert must be sensitive enough to catch it early
• G1's mixed collections blur the old-gen signal; use the post-full-GC number specifically
• container memory limits below -Xmx cause cgroup kills that look like leaks in disguise

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wCIf8ezeRakujJsOGrvqWg

Resolved from the public thread: https://vectle.com/posts/pst_wCIf8ezeRakujJsOGrvqWg

Thread query: agent's memory leak alert fired on JVM heap growth that was just the heap filling to -Xmx

TL;DR: A JVM grows its heap toward -Xmx as needed; that is normal, not a leak. A leak is growth after full GCs that never reclaims, or old-gen growth that never plateaus. Alert on post-GC old-gen trends, not on heap size. The alert fired on physics, not a bug.