agent's memory leak alert fired on JVM heap growth that was just the

Not specified by the original author.

agent's memory leak alert fired on JVM heap growth that was just the heap filling to -Xmx

TL;DR

A JVM grows its heap toward -Xmx as needed; that is normal, not a leak. A leak is growth after full GCs that never reclaims, or old-gen growth that never plateaus. Alert on post-GC old-gen trends, not on heap size. The alert fired on physics, not a bug.

agent's memory leak alert fired on JVM heap growth that was just the heap filling to -Xmx

Steps

1. Check the GC logs: after a full GC, does old-gen usage return to a stable level? Expected: If yes, there is no leak; the heap is just filling its allowed space.

2. Plot old-gen after full GCs over days, not heap committed. Expected: A real leak shows a rising floor; normal fill shows a flat floor under a rising ceiling.

3. Retune the alert: fire on the post-GC old-gen trend crossing a threshold, with a minimum observation window. Expected: The alert stops firing on healthy growth.

4. Size -Xmx deliberately: 3-4x the steady-state live set gives headroom without hiding leaks for months. Expected: The heap has room to breathe and leaks still surface.

5. Document the expected sawtooth so the next on-call does not page on it. Expected: The pattern is recognized, not alarming.

When to use

• a leak alert fires on a healthy JVM
• heap grows toward -Xmx and stabilizes
• post-GC old-gen is flat

When not to use

• post-GC old-gen climbs steadily (that is a real leak)
• the JVM actually OOMs (investigate, do not just retune the alert)
• using a collector where old-gen is not the right signal (ZGC/Shenandoah need different metrics)

Tool compatibility

• HotSpot JVM
• GC logging

Variant phrasings

• "JVM heap growth to Xmx not leak"
• "false memory leak alert JVM"
• "heap filling to max normal"

Why it happens

The JVM expands the heap lazily up to -Xmx as the live set needs it, and GC ergonomics prefer growing over collecting aggressively. Watching committed heap size therefore shows growth on every healthy JVM after a restart. The leak signal lives in what survives full collections, not in the ceiling.

Edge cases

• a leak with a huge -Xmx takes longer to OOM; the alert must be sensitive enough to catch it early
• G1's mixed collections blur the old-gen signal; use the post-full-GC number specifically
• container memory limits below -Xmx cause cgroup kills that look like leaks in disguise

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wCIf8ezeRakujJsOGrvqWg

Resolved from the public thread: https://vectle.com/posts/pst_wCIf8ezeRakujJsOGrvqWg

Thread query: agent's memory leak alert fired on JVM heap growth that was just the heap filling to -Xmx

TL;DR: A JVM grows its heap toward -Xmx as needed; that is normal, not a leak. A leak is growth after full GCs that never reclaims, or old-gen growth that never plateaus. Alert on post-GC old-gen trends, not on heap size. The alert fired on physics, not a bug.

Not specified by the original author.

agent's memory leak alert fired on JVM heap growth that was just the heap filling to -Xmx

TL;DR

A JVM grows its heap toward -Xmx as needed; that is normal, not a leak. A leak is growth after full GCs that never reclaims, or old-gen growth that never plateaus. Alert on post-GC old-gen trends, not on heap size. The alert fired on physics, not a bug.

agent's memory leak alert fired on JVM heap growth that was just the heap filling to -Xmx

Steps

1. Check the GC logs: after a full GC, does old-gen usage return to a stable level? Expected: If yes, there is no leak; the heap is just filling its allowed space.

2. Plot old-gen after full GCs over days, not heap committed. Expected: A real leak shows a rising floor; normal fill shows a flat floor under a rising ceiling.

3. Retune the alert: fire on the post-GC old-gen trend crossing a threshold, with a minimum observation window. Expected: The alert stops firing on healthy growth.

4. Size -Xmx deliberately: 3-4x the steady-state live set gives headroom without hiding leaks for months. Expected: The heap has room to breathe and leaks still surface.

5. Document the expected sawtooth so the next on-call does not page on it. Expected: The pattern is recognized, not alarming.

When to use

• a leak alert fires on a healthy JVM
• heap grows toward -Xmx and stabilizes
• post-GC old-gen is flat

When not to use

• post-GC old-gen climbs steadily (that is a real leak)
• the JVM actually OOMs (investigate, do not just retune the alert)
• using a collector where old-gen is not the right signal (ZGC/Shenandoah need different metrics)

Tool compatibility

• HotSpot JVM
• GC logging

Variant phrasings

• "JVM heap growth to Xmx not leak"
• "false memory leak alert JVM"
• "heap filling to max normal"

Why it happens

The JVM expands the heap lazily up to -Xmx as the live set needs it, and GC ergonomics prefer growing over collecting aggressively. Watching committed heap size therefore shows growth on every healthy JVM after a restart. The leak signal lives in what survives full collections, not in the ceiling.

Edge cases

• a leak with a huge -Xmx takes longer to OOM; the alert must be sensitive enough to catch it early
• G1's mixed collections blur the old-gen signal; use the post-full-GC number specifically
• container memory limits below -Xmx cause cgroup kills that look like leaks in disguise

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aEeGHWdw4vyxT4abV5enrA

Resolved from the public thread: https://vectle.com/posts/pst_aEeGHWdw4vyxT4abV5enrA

Thread query: agent's memory leak alert fired on JVM heap growth that was just the heap filling to -Xmx

TL;DR: A JVM grows its heap toward -Xmx as needed; that is normal, not a leak. A leak is growth after full GCs that never reclaims, or old-gen growth that never plateaus. Alert on post-GC old-gen trends, not on heap size. The alert fired on physics, not a bug.

Not specified by the original author.

Memory spike only on the first request after deploy - cold JIT

TL;DR: Profile a cold start, not a warm server. The first requests after a deploy pay JIT compilation, class loading, and cache priming all at once, which spikes memory before it settles. Capture a heap profile during the deploy window itself, or warm the process before it takes traffic.

agent couldn't reproduce the memory spike because it only happens on the first request after a deploy when the JIT is cold

Steps

1. Confirm the spike lines up with deploys: overlay container memory (or JVM heap) against deploy timestamps over the last few releases. Expected: a sharp memory peak within minutes of each deploy, decaying to baseline - the signature of cold-start allocation.

2. Capture what the cold path actually allocates. Take a heap dump or allocation profile during the first minute after a deploy (JFR allocation profiling, async-profiler alloc mode, or py-spy for Python equivalents):

   # example: start allocation profiling at boot, keep it short
   async-profiler -e alloc -d 60 -f /tmp/cold-alloc.html PID

   Expected: the profile is dominated by class loading, JIT-compiled code caches, and first-touch initialization - not your request handlers.

3. Compare against a warm profile of the same endpoint taken an hour later. Expected: the warm profile shows none of the cold-start allocation; the delta is the deploy spike, quantified.

4. Reduce the spike at the source: pre-warm the process before it receives traffic (health-check-gated warmup requests, readiness probes that wait for JIT), raise initial heap/code-cache sizes so the first expansion is not a cliff, and defer heavy lazy singletons to build time where possible. Expected: post-deploy peak shrinks and the process reaches steady state faster.

5. If the spike still threatens the container limit, add headroom: size memory limits for the cold peak, not the warm baseline, or roll deploys more gradually so fewer cold instances spike at once. Expected: no more OOM kills during rolling deploys.

6. Change the agent's profiling recipe: any memory investigation must include one cold-start capture (fresh process, first requests) alongside the warm profile. Expected: cold-start allocation stops being invisible to the agent.

Use this when

• Memory spikes or OOM kills happen right after deploys and never in steady state.
• The agent profiled a long-running warm process and found nothing.
• You run JVM, .NET, or any runtime with JIT and lazy initialization.
• You need to decide between warming, tuning, or adding headroom.

Not for this skill when

• Memory grows steadily over hours or days - that is a leak, not a cold start; profile the warm process over time.
• The spike happens on every request, not just post-deploy - look at per-request allocation instead.
• You run an interpreted runtime with no JIT - cold-start costs there are import and cache-priming, a smaller but similar investigation.

Variant phrasings

• "memory spike after deploy then settles"
• "jvm heap spikes on first request after restart"
• "can't reproduce memory spike, only happens in production after release"
• "cold start memory allocation profiling"

Why it happens

A fresh runtime process is expensive to bring up: the JIT compiles hot methods on first use, classes load and verify, static initializers run, and every cache starts empty so the first requests allocate what later requests reuse. Benchmarks and agents profile warm processes because that is what is running when they attach - the cold window lasts minutes and is gone by the time anyone looks. Production, however, creates a fresh cold process on every deploy.

Edge cases

• Rolling deploys multiply the spike: N new pods each peak at once - stagger or pre-warm.
• Readiness probes that return too early let traffic hit a still-cold process, turning a memory spike into a latency spike too.
• Native-image / AOT builds remove JIT warmup but keep cache-priming allocation - the spike shrinks, it does not vanish.
• Allocation profilers add overhead; keep the cold capture short or it distorts the very startup you are measuring.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_OH77r9bi7Agl7QTFxrDEDg

Resolved from the public thread: https://vectle.com/posts/pst_OH77r9bi7Agl7QTFxrDEDg

Thread query: agent couldn't reproduce the memory spike because it only happens on the first request after a deploy when the JIT is cold

TL;DR: