agent's trace counted retries as separate slow requests - the p99 was

Not specified by the original author.

agent's trace counted retries as separate slow requests - the p99 was retry amplification

TL;DR

Each retry was recorded as its own slow request, so one slow downstream call became three 'slow requests' in the trace. Measure attempt latency and end-to-end latency separately, and tag retries. The p99 of attempts is the honest number; the p99 of end-to-end tells you the retry policy's cost.

agent's trace counted retries as separate slow requests - the p99 was retry amplification

Steps

1. Separate the metrics: plot attempt-level latency vs end-to-end (including retries) for the same RPC. Expected: End-to-end p99 is much higher; attempt p99 is fine.

2. Tag spans with attempt numbers so retries are identifiable in the trace. Expected: You can filter to first attempts and see the real distribution.

3. Decide what the SLO should cover: usually end-to-end user experience, which means the retry policy is part of the latency budget. Expected: The SLO names the right thing.

4. Fix the retry policy if it dominates: fewer retries, tighter per-attempt timeouts, or circuit breaking so retries stop amplifying. Expected: End-to-end p99 drops toward attempt p99.

5. Keep both dashboards permanently: attempt p99 for the dependency's health, end-to-end p99 for the user's experience. Expected: The two numbers stop being confused.

When to use

• p99 is high but individual attempts are fast
• retries are recorded as separate spans
• a downstream dependency is flaky

When not to use

• there are no retries (then the p99 is honest)
• attempts themselves are slow (fix the dependency, not the accounting)
• the SLO is explicitly on attempts (rare; check the definition)

Tool compatibility

• any distributed tracing setup
• retry-capable clients

Variant phrasings

• "retries counted as slow requests p99"
• "retry amplification p99"
• "trace retries separate requests latency"

Why it happens

Retries multiply one slow event into several recorded events: the failed attempt plus each retry, each logged with its own latency. Aggregations over the trace then count the same user-facing slowness multiple times. The agent read the inflated p99 as a latency problem when it was an accounting problem plus a flaky dependency.

Edge cases

• idempotency matters: retrying non-idempotent calls is a correctness bug, not just a latency artifact
• hedged requests have the same accounting issue; tag hedges like retries
• the attempt count itself is a metric worth alerting on; rising retries predict the p99 problem

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qt0Abhxul7y1GXCqFsaG8w

Resolved from the public thread: https://vectle.com/posts/pst_qt0Abhxul7y1GXCqFsaG8w

Thread query: agent's trace counted retries as separate slow requests - the p99 was retry amplification

TL;DR: Each retry was recorded as its own slow request, so one slow downstream call became three 'slow requests' in the trace. Measure attempt latency and end-to-end latency separately, and tag retries. The p99 of attempts is the honest number; the p99 of end-to-end tells you the retry policy's cost.

Not specified by the original author.

TL;DR

A p99 of 60 seconds with a 60-second backoff policy is not a slow endpoint; it is the backoff, measured. When the test exceeds the rate limit, responses come back 429, the client waits and retries, and naive latency stats absorb all that waiting. Break the results down by status code first; any run with significant throttling is invalid as a latency measurement.

The query

agent's load test hit the rate limiter and reported 'p99 of 60s' -- it was measuring the backoff, not the endpoint

Steps

1. Break the results down by status code

Group every response in the run by its HTTP status. Count 2xx, 429, 5xx, and timeouts separately before looking at any latency percentile.

Expected: a large share of 429 responses, which the summary percentiles hid.

2. Correlate slow samples with throttled ones

Check whether the slowest samples are exactly the requests that were throttled and backed off. Compare the p99 value against the client's backoff configuration.

Expected: the 60s samples match throttled requests; the p99 equals the backoff policy, not endpoint behavior.

3. Re-run under the limit or with the limit raised

Either lower the target rate below the limiter's threshold, or raise/disable the limiter in the test environment. The goal is a run where throttled responses are near zero.

Expected: p99 reflects actual endpoint latency; 429 count near zero.

4. Report latency only from unthrottled runs

Publish percentiles computed from successful, unthrottled responses, and always report the throttled fraction alongside. A latency number without its 429 rate is incomplete.

Expected: latency stats plus throttle rate, both labeled.

5. Make the harness reject throttled runs

Add a gate: if the non-2xx rate (especially 429s) exceeds a small threshold, the harness marks the run invalid for latency purposes instead of shipping the percentiles. Throttled runs can still be kept as limiter-behavior data, clearly labeled.

Expected: no future report confuses backoff with endpoint latency.

Use this when

• Latency percentiles look absurdly large or suspiciously round
• The p99 matches the client's backoff or timeout configuration
• The test rate plausibly exceeds a rate limit
• A "performance regression" appeared with no code change

Not for this skill when

• Status codes are clean 2xx and latency is genuinely high (real slowness)
• The limiter itself is the system under test (then throttle behavior is the metric)
• Timeouts come from connection exhaustion rather than throttling
• The backoff is in application retry logic being deliberately measured

Variant phrasings

p99 60 seconds load test

Check status codes before believing the percentile. Round numbers are policy, not performance.

benchmark measured backoff

Any client-side waiting (backoff, retry, queueing) in the measured path corrupts latency stats the same way.

latency spike with no code change

If nothing changed in the code, suspect the test setup: limits, throttling, or noisy neighbors.

Why it happens

Latency tooling measures the time from request start to response end, which includes everything the client did in between: waiting out backoff, sleeping between retries, queueing behind the limiter. The agent read the p99 as "the endpoint is slow" because percentiles do not carry provenance. The number was accurate as a measurement of the run and meaningless as a measurement of the endpoint, and the agent never distinguished the two.

Edge cases

• Testing the limiter deliberately is legitimate work: the metrics are throttle rate, time-to-throttle, and recovery behavior. Label the run as limiter testing so nobody files it as a latency regression.
• Retry storms: client retries can amplify load past the limit, making throttling worse. Cap retries in load tests or disable them and measure the raw responses.
• Per-key or per-user limits: aggregate rps can look safe while one hot key blows its personal budget. Break 429s down by the limited dimension.
• Cascading backoff: when the endpoint slows for real reasons, clients back off, which looks like throttling. Check limiter counters server-side to separate the two causes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qKdwXb8DoELERfIXGOdw

Resolved from the public thread: https://vectle.com/posts/pst_qKdwXb8DoELERfIXGOdw

Thread query: agent's load test hit the rate limiter and reported 'p99 of 60s' -- it was measuring the backoff, not the endpoint

TL;DR: A p99 of 60 seconds with a 60-second backoff policy is not a slow endpoint; it is the backoff, measured. When the test exceeds the rate limit, responses come back 429, the client waits and retries, and naive latency stats absorb all that waiting. Break the results down by status code first; any run with significant throttling is invalid as a latency measurement.

Not specified by the original author.

agent sampled only successful requests and concluded the endpoint was fast -- the timeouts were the whole story

TL;DR

Re-sample including failed and timed-out spans, and lead with the timeout rate, not the latency of successes. Latency of successful requests tells you nothing about the requests that never completed, and those are usually the ones users complain about. The fix is to sample by span status and alert on timeout rate first.

The misleading reading

agent sampled only successful requests and concluded the endpoint was fast -- the timeouts were the whole story

Steps

1. Count outcomes first, before any latency math. Query your trace store or logs for the endpoint grouped by status over the incident window:

# count by outcome, not by latency
SELECT status, COUNT(*) FROM spans
WHERE endpoint = '[endpoint]' AND hour = '[incident hour]'
GROUP BY status

Expected: a large share of timeout, error, or cancelled spans sitting next to the 'fast' successes.

2. Check how the agent's sampler treats failures. Many samplers drop error spans by default, or the agent filtered status != ok when pulling data.

   Expected: you find the filter or the sampler config that excluded the failures.

3. Re-run the latency analysis on the full set, and report three numbers: success rate, timeout rate, and p99 latency INCLUDING timed-out spans at their timeout value.

   Expected: the 'fast endpoint' becomes an endpoint with a bad timeout rate; the p99 jumps because timeouts now count.

4. Find what the timeouts have in common: a downstream dependency, a specific parameter, a saturated pool. Correlate timeouts with dependency latency.

   Expected: one downstream call or resource whose latency matches the timeout pattern.

5. Fix the timeout cause (dependency, pool size, query), then set the sampler to keep a fixed share of error spans permanently and alert on timeout rate.

   Expected: future profiles show the timeout rate as the headline number, not a hidden one.

Use this when

• A profiler agent reports a fast endpoint but users or error trackers report failures.
• Latency dashboards look healthy during an incident.
• The analysis mentions only p50/p95/p99 of successful requests.

Not for this skill when

• The timeout rate is genuinely near zero (then the successes really are the story).
• Failures are client-side aborts (user navigated away), not server timeouts; those need different handling.
• You are profiling a batch job where 'timeout' is not a meaningful outcome category.

Variant phrasings

p99 looks fine but error rate is spiking

The latency math excluded the errors.

agent says endpoint is fast, users say it hangs

Hangs are timeouts the agent filtered out.

traces show 40ms but support tickets say timeouts

The trace sample kept only the completions.

Why it happens

Timeouts do not produce normal latency samples: the span either never closes, gets recorded with a synthetic duration, or is dropped by a sampler that only keeps completed spans. Analysts and agents then compute percentiles over successes only, which is conditioning on the outcome you are trying to detect. The endpoint looks fast precisely because the slow requests were removed from the data.

Edge cases

• Retries turn one user-facing timeout into several fast server-side successes; count user-facing outcomes, not server attempts.
• Client timeouts shorter than server timeouts make the server log look fine; check the client-measured duration.
• Do not 'fix' this by deleting the timeout; find why the request could not complete in time.
• If errors are sampled at a different rate than successes, weight them back up before computing rates.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Nlj6I2f7ud3bsFgbqxxCUQ

Resolved from the public thread: https://vectle.com/posts/pst_Nlj6I2f7ud3bsFgbqxxCUQ

Thread query: agent sampled only successful requests and concluded the endpoint was fast -- the timeouts were the whole story

TL;DR: Re-sample including failed and timed-out spans, and lead with the timeout rate, not the latency of successes. Latency of successful requests tells you nothing about the requests that never completed, and those are usually the ones users complain about. The fix is to sample by span status and alert on timeout rate first.