agent measured an 'improvement' after removing the profiler -- the real fix was uninstalling the measurement

TL;DR

There was no improvement. You compared a profiled run against an unprofiled run, and the "faster" variant was just the one with the measurement turned off. Re-run the comparison with identical instrumentation on both sides. If the gap disappears, the profiler was the whole story and the code never changed performance at all.

agent measured an 'improvement' after removing the profiler -- the real fix was uninstalling the measurement

1. Spot the broken comparison. Check how each side of the benchmark was measured: profiler on for the "before," profiler off for the "after." Expected: the instrumentation differs between the two runs.
2. Re-run both sides identically. Either profile both variants or profile neither, keeping everything else the same. Expected: the "improvement" shrinks to zero or to the profiler's known overhead.
3. Measure the overhead separately. Run the same code with the profiler on and off and record the delta. Expected: the delta matches the supposed improvement almost exactly.
4. Write down the rule. Benchmarks in this codebase always use identical instrumentation on both arms, and any reported win must survive the profiler-on-versus-off check. Expected: nobody ships a "profiler removal" as a performance fix again.

Use this when

• A performance win appeared right after a profiler or agent was removed or disabled
• The before and after runs used different measurement setups
• The "optimized" code is byte-identical to the old code

Not for this skill when

• Both sides were measured identically and the win persists: that is a real improvement
• The change also altered code: then isolate the code change from the instrumentation change before claiming anything

Variant phrasings

• "app got faster after removing APM agent is that real"
• "benchmark shows improvement but I changed the profiler config too"
• "how to tell if a speedup is just measurement overhead"

Why it happens

A benchmark measures the system under test plus the measurement apparatus. When the apparatus changes between runs, its cost shows up as a difference in the results. Removing a profiler removes its per-request cost, which looks exactly like making the app faster, down to the shape of the latency curve.

Edge cases

• Partial changes count too: lowering the sample rate between runs creates the same illusion at smaller scale
• The reverse happens as well: adding a profiler for the "after" run can hide a real improvement
• Overhead is often uneven across endpoints, so the fake win can look oddly specific to some routes

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hISY7vlUETH47i5RurAtOw