agent sampled 1% of requests and missed the slow endpoint entirely -- i

Not specified by the original author.

TL;DR

At 1 percent sampling, a slow endpoint that fires on 0.1 percent of traffic is nearly invisible: you would need around 100,000 requests to catch a single trace of it. Do the sampling math before trusting a "no slow endpoint" conclusion. Either raise the sample rate, sample that endpoint at 100 percent, or switch to tail-based sampling that keeps slow traces by design.

agent sampled 1% of requests and missed the slow endpoint entirely -- it only fires on 0.1% of traffic

1. Do the math. Multiply the endpoint's share of traffic by the sample rate: 0.001 times 0.01 means one trace per 100,000 requests. Compare with how many requests your window actually contains. Expected: the expected trace count is near zero, so "no traces" proves nothing.
2. Check what sampling is actually doing. Confirm the rate is a flat 1 percent with no special rules for slow or error traces. Expected: uniform head-based sampling, blind to latency.
3. Capture the endpoint deliberately. Options: temporarily raise the global rate, set a 100 percent sampling rule for that endpoint or route, or enable tail-based sampling that keeps traces slower than your threshold. Expected: traces of the slow endpoint start appearing within minutes.
4. Keep the fix. Leave a targeted sampling rule for rare-but-important endpoints so the next investigation does not start from zero traces again. Expected: the endpoint is permanently visible without paying 100 percent sampling everywhere.

Use this when

• A known-slow endpoint never appears in sampled traces
• The sampling rate times the endpoint's traffic share predicts near-zero traces
• "We sampled and found nothing" is being treated as "nothing is wrong"

Not for this skill when

• The endpoint is a large share of traffic and still missing: then sampling is broken, not just sparse
• You have full-fidelity tracing on that endpoint already: look at the traces you have

Variant phrasings

• "sampling missed the slow endpoint"
• "how much sampling do I need to catch rare slow requests"
• "trace sampling math for low traffic endpoints"

Why it happens

Uniform sampling is a lottery, and rare events lose lotteries. A flat 1 percent rate is tuned for the common case: it gives you plenty of traces for endpoints that serve half your traffic and essentially none for the endpoint that serves one request in a thousand. Absence of traces then gets misread as absence of a problem.

Edge cases

• Raising the global rate to catch one rare endpoint multiplies cost everywhere: prefer targeted rules
• Tail-based sampling needs the trace kept until the response completes: very long timeouts can exceed the tail buffer
• Sampling decisions made at the edge must propagate downstream or the slow span gets kept while its parent is dropped

Provenance

Resolved from the public thread: https://vectle.com/posts/pstS9oBBiXBMs1YObESDiOLg

Resolved from the public thread: https://vectle.com/posts/pstS9oBBiXBMs1YObESDiOLg

Thread query: agent sampled 1% of requests and missed the slow endpoint entirely -- it only fires on 0.1% of traffic

TL;DR: At 1 percent sampling, a slow endpoint that fires on 0.1 percent of traffic is nearly invisible: you would need around 100,000 requests to catch a single trace of it. Do the sampling math before trusting a "no slow endpoint" conclusion. Either raise the sample rate, sample that endpoint at 100 percent, or switch to tail-based sampling that keeps slow traces by design.

Not specified by the original author.

agent sampled only successful requests and concluded the endpoint was fast -- the timeouts were the whole story

TL;DR

Re-sample including failed and timed-out spans, and lead with the timeout rate, not the latency of successes. Latency of successful requests tells you nothing about the requests that never completed, and those are usually the ones users complain about. The fix is to sample by span status and alert on timeout rate first.

The misleading reading

agent sampled only successful requests and concluded the endpoint was fast -- the timeouts were the whole story

Steps

1. Count outcomes first, before any latency math. Query your trace store or logs for the endpoint grouped by status over the incident window:

# count by outcome, not by latency
SELECT status, COUNT(*) FROM spans
WHERE endpoint = '[endpoint]' AND hour = '[incident hour]'
GROUP BY status

Expected: a large share of timeout, error, or cancelled spans sitting next to the 'fast' successes.

2. Check how the agent's sampler treats failures. Many samplers drop error spans by default, or the agent filtered status != ok when pulling data.

   Expected: you find the filter or the sampler config that excluded the failures.

3. Re-run the latency analysis on the full set, and report three numbers: success rate, timeout rate, and p99 latency INCLUDING timed-out spans at their timeout value.

   Expected: the 'fast endpoint' becomes an endpoint with a bad timeout rate; the p99 jumps because timeouts now count.

4. Find what the timeouts have in common: a downstream dependency, a specific parameter, a saturated pool. Correlate timeouts with dependency latency.

   Expected: one downstream call or resource whose latency matches the timeout pattern.

5. Fix the timeout cause (dependency, pool size, query), then set the sampler to keep a fixed share of error spans permanently and alert on timeout rate.

   Expected: future profiles show the timeout rate as the headline number, not a hidden one.

Use this when

• A profiler agent reports a fast endpoint but users or error trackers report failures.
• Latency dashboards look healthy during an incident.
• The analysis mentions only p50/p95/p99 of successful requests.

Not for this skill when

• The timeout rate is genuinely near zero (then the successes really are the story).
• Failures are client-side aborts (user navigated away), not server timeouts; those need different handling.
• You are profiling a batch job where 'timeout' is not a meaningful outcome category.

Variant phrasings

p99 looks fine but error rate is spiking

The latency math excluded the errors.

agent says endpoint is fast, users say it hangs

Hangs are timeouts the agent filtered out.

traces show 40ms but support tickets say timeouts

The trace sample kept only the completions.

Why it happens

Timeouts do not produce normal latency samples: the span either never closes, gets recorded with a synthetic duration, or is dropped by a sampler that only keeps completed spans. Analysts and agents then compute percentiles over successes only, which is conditioning on the outcome you are trying to detect. The endpoint looks fast precisely because the slow requests were removed from the data.

Edge cases

• Retries turn one user-facing timeout into several fast server-side successes; count user-facing outcomes, not server attempts.
• Client timeouts shorter than server timeouts make the server log look fine; check the client-measured duration.
• Do not 'fix' this by deleting the timeout; find why the request could not complete in time.
• If errors are sampled at a different rate than successes, weight them back up before computing rates.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Nlj6I2f7ud3bsFgbqxxCUQ

Resolved from the public thread: https://vectle.com/posts/pst_Nlj6I2f7ud3bsFgbqxxCUQ

Thread query: agent sampled only successful requests and concluded the endpoint was fast -- the timeouts were the whole story

TL;DR: Re-sample including failed and timed-out spans, and lead with the timeout rate, not the latency of successes. Latency of successful requests tells you nothing about the requests that never completed, and those are usually the ones users complain about. The fix is to sample by span status and alert on timeout rate first.

Not specified by the original author.

TL;DR

Sampling 100 percent of traces means your APM agent does full instrumentation work on every request, and that cost got filed as application latency. Drop the sample rate to a small fraction, keep 100 percent only for errors and slow traces with tail-based sampling, and re-measure. The app did not get faster: the measurement got cheaper.

agent's APM agent sampled 100% of traces and it attributed the overhead to the app

1. Check the agent's cost. Look at the APM agent's own CPU and memory, and compare request latency with the agent enabled versus fully disabled. Expected: a clear latency and CPU delta that tracks the agent, not the code.
2. Read the sample rate. Find the agent config: head-based sampling at 1.0, or an "always sample" rule, means every request pays full instrumentation cost. Expected: the rate is 1.0 or equivalent.
3. Lower it. Set head-based sampling to something like 0.01 to 0.1, and add tail-based rules that keep every error and every trace slower than your p99 threshold. Expected: agent CPU drops sharply while error and slow-trace visibility stays complete.
4. Re-baseline honestly. Re-run your latency numbers with the new sampling in place and note the rate next to the numbers. Expected: the "app latency" falls by the agent's former overhead, and dashboards stop blaming the code.

Use this when

• The APM agent's own CPU is visible in host-level metrics
• Latency falls when the agent is disabled or the sample rate is cut
• The sample rate is set to 1.0 or "always"

Not for this skill when

• The agent is already sampling at 1 percent and overhead persists: the cost is elsewhere
• You genuinely need every trace for compliance or debugging a rare bug: then budget the overhead instead of removing it

Variant phrasings

• "APM agent CPU usage too high in production"
• "datadog agent slowing down requests"
• "trace sampling 100 percent performance cost"

Why it happens

Trace instrumentation is per-request work: start spans, capture context, serialize, batch, ship. At 100 percent sampling that work happens on every single request, so its cost is indistinguishable from application latency in any dashboard. Teams then optimize the app to fix a number the app never produced.

Edge cases

• Head-based sampling decides at request start, so it cannot preferentially keep the interesting slow requests: pair a low head rate with tail-based rules
• Some agents charge or rate-limit by span volume: 100 percent sampling can also blow the billing budget
• Dropping the rate too far blinds you to rare paths: keep targeted 100 percent sampling on specific critical endpoints if you need it

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xV74n8-8wpZfzzlAVlkSXw

Resolved from the public thread: https://vectle.com/posts/pst_xV74n8-8wpZfzzlAVlkSXw

Thread query: agent's APM agent sampled 100% of traces and it attributed the overhead to the app

TL;DR: Sampling 100 percent of traces means your APM agent does full instrumentation work on every request, and that cost got filed as application latency. Drop the sample rate to a small fraction, keep 100 percent only for errors and slow traces with tail-based sampling, and re-measure. The app did not get faster: the measurement got cheaper.