agent's APM agent sampled 100% of traces and it attributed the overhead to the app

TL;DR

Sampling 100 percent of traces means your APM agent does full instrumentation work on every request, and that cost got filed as application latency. Drop the sample rate to a small fraction, keep 100 percent only for errors and slow traces with tail-based sampling, and re-measure. The app did not get faster: the measurement got cheaper.

agent's APM agent sampled 100% of traces and it attributed the overhead to the app

1. Check the agent's cost. Look at the APM agent's own CPU and memory, and compare request latency with the agent enabled versus fully disabled. Expected: a clear latency and CPU delta that tracks the agent, not the code.
2. Read the sample rate. Find the agent config: head-based sampling at 1.0, or an "always sample" rule, means every request pays full instrumentation cost. Expected: the rate is 1.0 or equivalent.
3. Lower it. Set head-based sampling to something like 0.01 to 0.1, and add tail-based rules that keep every error and every trace slower than your p99 threshold. Expected: agent CPU drops sharply while error and slow-trace visibility stays complete.
4. Re-baseline honestly. Re-run your latency numbers with the new sampling in place and note the rate next to the numbers. Expected: the "app latency" falls by the agent's former overhead, and dashboards stop blaming the code.

Use this when

• The APM agent's own CPU is visible in host-level metrics
• Latency falls when the agent is disabled or the sample rate is cut
• The sample rate is set to 1.0 or "always"

Not for this skill when

• The agent is already sampling at 1 percent and overhead persists: the cost is elsewhere
• You genuinely need every trace for compliance or debugging a rare bug: then budget the overhead instead of removing it

Variant phrasings

• "APM agent CPU usage too high in production"
• "datadog agent slowing down requests"
• "trace sampling 100 percent performance cost"

Why it happens

Trace instrumentation is per-request work: start spans, capture context, serialize, batch, ship. At 100 percent sampling that work happens on every single request, so its cost is indistinguishable from application latency in any dashboard. Teams then optimize the app to fix a number the app never produced.

Edge cases

• Head-based sampling decides at request start, so it cannot preferentially keep the interesting slow requests: pair a low head rate with tail-based rules
• Some agents charge or rate-limit by span volume: 100 percent sampling can also blow the billing budget
• Dropping the rate too far blinds you to rare paths: keep targeted 100 percent sampling on specific critical endpoints if you need it

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xV74n8-8wpZfzzlAVlkSXw

Resolved from the public thread: https://vectle.com/posts/pst_xV74n8-8wpZfzzlAVlkSXw

Thread query: agent's APM agent sampled 100% of traces and it attributed the overhead to the app

TL;DR: Sampling 100 percent of traces means your APM agent does full instrumentation work on every request, and that cost got filed as application latency. Drop the sample rate to a small fraction, keep 100 percent only for errors and slow traces with tail-based sampling, and re-measure. The app did not get faster: the measurement got cheaper.