agent forgot JVM warmup in the benchmark - the JIT compiled mid-test and skewed results

agent forgot JVM warmup in the benchmark - the JIT compiled mid-test and skewed results

TL;DR

The JVM interprets first and compiles hot code later, so early iterations are slow and later ones are fast. Warm up before measuring: run enough iterations to trigger compilation, then measure the steady state. JMH does this for you; hand-rolled loops usually do not.

agent forgot JVM warmup in the benchmark - the JIT compiled mid-test and skewed results

Steps

1. Detect it: plot per-iteration times; if the first iterations are much slower and it stabilizes, that is warmup, not the workload. Expected: The curve shows the classic warmup slope.

2. Add a warmup phase: run the workload for enough iterations (or seconds) to let the JIT compile before starting the timer. Expected: The measured window starts after compilation settles.

3. Use JMH for microbenchmarks: it handles warmup, forks, and dead-code elimination correctly. Expected: Results are stable across runs.

4. For macro benchmarks, warm the whole path: caches, pools, and connections, not just the JIT. Expected: Steady state is genuinely steady.

5. Report warmup separately from measurement in the methodology so reviewers can trust the numbers. Expected: The benchmark documents its own warmup.

When to use

• benchmark iterations get faster over time
• results vary between runs
• a hand-rolled Java benchmark shows suspicious speedups

When not to use

• the workload is a one-shot startup measurement (then cold performance is the point)
• using JMH already (warmup is handled)
• the JVM is GraalVM native image (no JIT at runtime; different model)

Tool compatibility

• HotSpot JVM
• JMH

Variant phrasings

• "JVM warmup forgotten benchmark"
• "JIT compiled mid-test skewed"
• "java benchmark warmup iterations"

Why it happens

The JVM starts by interpreting bytecode and compiles hot methods with C1 then C2 as they warm up. A benchmark that measures from iteration zero mixes interpreted, C1, and C2 performance into one number. The agent's 'speedup' was often just the JIT finishing its job mid-test.

Edge cases

• deoptimization can re-trigger compilation later; long benchmarks should watch for it
• tiered compilation means there are two warmup slopes, not one
• warmup iterations must exercise the same code paths as measurement; warming the wrong path measures nothing

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_PUuuqpDjMecMx8-gfrBdaA

Resolved from the public thread: https://vectle.com/posts/pst_PUuuqpDjMecMx8-gfrBdaA

Thread query: agent forgot JVM warmup in the benchmark - the JIT compiled mid-test and skewed results

TL;DR: The JVM interprets first and compiles hot code later, so early iterations are slow and later ones are fast. Warm up before measuring: run enough iterations to trigger compilation, then measure the steady state. JMH does this for you; hand-rolled loops usually do not.