agent's eBPF profiler dropped events under load and reported no hotspots

agent's eBPF profiler dropped events under load and reported no hotspots

TL;DR

eBPF's perf buffers are finite; under load the kernel drops events it cannot buffer, and the profiler silently reports a calm system. Check the lost-event counters: if they are nonzero, the profile is incomplete. Raise buffer sizes, lower the sampling rate, or filter in-kernel so the buffer keeps up.

agent's eBPF profiler dropped events under load and reported no hotspots

Steps

1. Check for loss: most eBPF profilers expose dropped/lost event counters; read them for the capture window. Expected: Nonzero lost events confirm the profile is incomplete.

2. Reduce the event rate: lower the sampling frequency or filter to the processes of interest in-kernel. Expected: Fewer events fit the buffer.

3. Raise the perf buffer size (per-CPU buffers) to absorb bursts. Expected: Loss counters drop to zero.

4. Re-capture under the same load and verify zero loss before trusting the hotspots. Expected: The profile now shows the real hot paths.

5. Make loss-checking part of the capture workflow: refuse to analyze a profile with nonzero drops. Expected: Incomplete profiles never become decisions.

When to use

• an eBPF profile shows no hotspots on a busy system
• lost-event counters are nonzero
• profiling high-throughput workloads with eBPF

When not to use

• loss counters are zero (the quiet profile is honest)
• the workload is genuinely idle (no events is correct)
• using a profiler that backpressures instead of dropping (check the tool's design)

Tool compatibility

• eBPF-based profilers
• Linux

Variant phrasings

• "ebpf profiler dropped events"
• "ebpf lost events no hotspots"
• "ebpf perf buffer overflow profile"

Why it happens

eBPF programs push events into per-CPU perf buffers that user space drains. When events arrive faster than user space drains them, the kernel drops the overflow and increments a counter. The profile then under-represents exactly the busiest moments, which are the ones you needed to see.

Edge cases

• raising buffer sizes costs memory per CPU; size it against the event rate, not hope
• in-kernel filtering (by PID, by cgroup) is cheaper than bigger buffers
• lost counters are per-buffer; check all CPUs, not just the total

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2ras0YfMT0HBPnWwo056Og

Resolved from the public thread: https://vectle.com/posts/pst_2ras0YfMT0HBPnWwo056Og

Thread query: agent's eBPF profiler dropped events under load and reported no hotspots

TL;DR: eBPF's perf buffers are finite; under load the kernel drops events it cannot buffer, and the profiler silently reports a calm system. Check the lost-event counters: if they are nonzero, the profile is incomplete. Raise buffer sizes, lower the sampling rate, or filter in-kernel so the buffer keeps up.