agent profiled with debug symbols on and optimized-away hot paths looked cold

TL;DR

You profiled a debug build, where the compiler barely optimizes, and the hot paths that get inlined away in production showed up as cold. Profile the build you ship: keep optimization on and add debug info alongside it, plus frame pointers for clean stacks. Then compare: the production hot spots will look nothing like the debug-build profile.

agent profiled with debug symbols on and optimized-away hot paths looked cold

1. Check what you actually profiled. Look at the build flags: -O0 or a debug preset means the compiler did not inline, unroll, or eliminate anything. Expected: the profiled binary was built without optimization.
2. Rebuild the way production builds. Use -O2 (or your release setting) with -g for debug info and -fno-omit-frame-pointer for reliable stacks. Expected: optimization stays on, symbols and frame pointers make the profile readable.
3. Re-run the profile on the optimized binary under the same workload. Expected: the hot functions change dramatically; small functions vanish into their callers through inlining, which is exactly what production does.
4. Make it the default. Wire the profiling build mode into your tooling so nobody profiles -O0 again: a dedicated profile build type with release optimization plus symbols. Expected: future profiles match production behavior.

Use this when

• A flame graph from a dev or debug build contradicts production measurements
• Tiny helper functions dominate the profile but production does not show them
• The profiled binary was built with -O0 or a debug preset

Not for this skill when

• The profile already came from an optimized build: then the hot paths are real
• You are debugging correctness, not performance: debug builds are fine for that

Variant phrasings

• "flame graph different in debug vs release build"
• "hot path disappeared after compiler optimization"
• "should I profile debug or release builds"

Why it happens

Optimization rewrites the code the profiler sees: inlining dissolves small functions into their callers, dead code disappears, loops get unrolled and vectorized. A debug build preserves the source-shaped call tree, so the profile describes a program that production never runs. Debug symbols alone are harmless; it is the missing optimization that lies.

Edge cases

• -g with -O2 can still confuse line-level attribution after aggressive inlining: trust function-level data more than line-level
• Some profilers need frame pointers: without -fno-omit-frame-pointer on x86-64 you get broken stacks even on optimized builds
• Link-time optimization moves code across translation units: profile the final linked binary, not object files

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1V5lIr5Zpnp3kxghWFTggA