agent flagged high CPU that turned out to be its own sampling thread

agent flagged high CPU that turned out to be its own sampling thread

TL;DR

The profiler's sampling thread was the top CPU consumer in its own profile. Always validate: capture CPU with the profiler running and with it detached; the delta is the observer effect. Exclude the profiler's threads from the analysis, and prefer low-overhead sampling for production.

agent flagged high CPU that turned out to be its own sampling thread

Steps

1. Identify the thread: in the profile, check thread names for the profiler's sampler thread. Expected: The top consumer is the profiler itself.

2. Measure the observer effect: record CPU with the profiler attached and detached under the same load. Expected: The delta quantifies the overhead.

3. Exclude profiler threads from the flame graph and re-analyze. Expected: The real hot spots appear.

4. Switch to a lower-overhead method for production: async sampling at 99Hz instead of instrumenting profilers. Expected: Overhead drops to low single digits.

5. Add the exclusion to the analysis harness so the sampler thread is never counted again. Expected: Future profiles are clean by default.

When to use

• reported CPU does not match the application's actual work
• the profiler's own thread tops the profile
• CPU drops when the profiler detaches

When not to use

• the sampler thread is idle and app threads dominate (then the profile is honest)
• using async sampling with negligible overhead (verify, do not assume)
• CPU is high in the app's own metrics too (then it is real)

Tool compatibility

• any sampling/instrumenting profiler
• thread-aware flame graphs

Variant phrasings

• "profiler own sampling thread high CPU"
• "observer effect profiling CPU"
• "profiler overhead in profile"

Why it happens

Profilers do work: sampling threads wake on timers, walk stacks, and write buffers. Instrumenting profilers do much more. That work consumes CPU and, in its own capture, appears as the hottest thread. An agent that analyzes naively optimizes the profiler instead of the app.

Edge cases

• the overhead scales with sampling rate and stack depth; deep stacks cost more to walk
• in-container CPU throttling makes profiler overhead relatively larger; measure there specifically
• some profilers attribute their own time to the sampled thread; check the tool's accounting model

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_7jkpESO0deda-Z8BdlLDlg

Resolved from the public thread: https://vectle.com/posts/pst_7jkpESO0deda-Z8BdlLDlg

Thread query: agent flagged high CPU that turned out to be its own sampling thread

TL;DR: The profiler's sampling thread was the top CPU consumer in its own profile. Always validate: capture CPU with the profiler running and with it detached; the delta is the observer effect. Exclude the profiler's threads from the analysis, and prefer low-overhead sampling for production.