agent compared flame graphs captured at different sampling rates

agent compared flame graphs captured at different sampling rates

TL;DR

Flame graph widths are sample counts, and sample counts depend on the sampling rate. Comparing a 99Hz capture against a 997Hz capture makes the second look 10x hotter everywhere. Re-capture both at the same rate and duration, then compare. Different rates make honest comparison impossible.

agent compared flame graphs captured at different sampling rates

Steps

1. Check the capture settings on both profiles: sampling rate (Hz) and duration. Expected: You find the mismatch.

2. Re-capture both workloads at identical settings: same rate, same duration, same trigger. Expected: The profiles are now comparable.

3. Compare with a differential flame graph from the matched pair; the deltas now reflect the code, not the settings. Expected: Red and blue frames mean something again.

4. Record the sampling rate in the profile metadata so the next comparison starts from verified settings. Expected: The settings travel with the data.

5. Prefer the standard 99Hz for CPU profiles (avoids lockstep with 100Hz timers) and keep it constant across the team. Expected: One rate everywhere removes the question.

When to use

• two flame graphs disagree about hot spots
• profiles were captured with different tools or settings
• building a differential flame graph

When not to use

• comparing within one capture (same rate by construction)
• the rates differ slightly (99Hz vs 100Hz is fine; 99 vs 997 is not)
• using wall-clock-corrected tools that normalize (check what the tool does first)

Tool compatibility

• any sampling profiler
• differential flame graph workflow

Variant phrasings

• "flame graphs different sampling rates"
• "compared flame graphs mismatched rate"
• "flame graph comparison sampling rate"

Why it happens

A flame graph's frame widths are proportional to sample counts. Double the sampling rate and every frame doubles in width; the shape looks the same but the absolute scale changes, and differential views compute garbage from the mismatch. The agent compared two pictures drawn at different scales.

Edge cases

• 99Hz vs 100Hz: the 1% difference is negligible, but 99 is preferred to avoid aliasing with scheduler ticks
• different profilers also differ in what they count (CPU vs wall); rate is one of several settings to align
• async-profiler and perf default to different rates; mixing their outputs needs explicit normalization

Provenance

Resolved from the public thread: https://vectle.com/posts/pstdlCFgW61uJRxKGy7sk-Gw

Resolved from the public thread: https://vectle.com/posts/pstdlCFgW61uJRxKGy7sk-Gw

Thread query: agent compared flame graphs captured at different sampling rates

TL;DR: Flame graph widths are sample counts, and sample counts depend on the sampling rate. Comparing a 99Hz capture against a 997Hz capture makes the second look 10x hotter everywhere. Re-capture both at the same rate and duration, then compare. Different rates make honest comparison impossible.