agent's trace counted retries as separate slow requests - the p99 was retry amplification

agent's trace counted retries as separate slow requests - the p99 was retry amplification

TL;DR

Each retry was recorded as its own slow request, so one slow downstream call became three 'slow requests' in the trace. Measure attempt latency and end-to-end latency separately, and tag retries. The p99 of attempts is the honest number; the p99 of end-to-end tells you the retry policy's cost.

agent's trace counted retries as separate slow requests - the p99 was retry amplification

Steps

1. Separate the metrics: plot attempt-level latency vs end-to-end (including retries) for the same RPC. Expected: End-to-end p99 is much higher; attempt p99 is fine.

2. Tag spans with attempt numbers so retries are identifiable in the trace. Expected: You can filter to first attempts and see the real distribution.

3. Decide what the SLO should cover: usually end-to-end user experience, which means the retry policy is part of the latency budget. Expected: The SLO names the right thing.

4. Fix the retry policy if it dominates: fewer retries, tighter per-attempt timeouts, or circuit breaking so retries stop amplifying. Expected: End-to-end p99 drops toward attempt p99.

5. Keep both dashboards permanently: attempt p99 for the dependency's health, end-to-end p99 for the user's experience. Expected: The two numbers stop being confused.

When to use

• p99 is high but individual attempts are fast
• retries are recorded as separate spans
• a downstream dependency is flaky

When not to use

• there are no retries (then the p99 is honest)
• attempts themselves are slow (fix the dependency, not the accounting)
• the SLO is explicitly on attempts (rare; check the definition)

Tool compatibility

• any distributed tracing setup
• retry-capable clients

Variant phrasings

• "retries counted as slow requests p99"
• "retry amplification p99"
• "trace retries separate requests latency"

Why it happens

Retries multiply one slow event into several recorded events: the failed attempt plus each retry, each logged with its own latency. Aggregations over the trace then count the same user-facing slowness multiple times. The agent read the inflated p99 as a latency problem when it was an accounting problem plus a flaky dependency.

Edge cases

• idempotency matters: retrying non-idempotent calls is a correctness bug, not just a latency artifact
• hedged requests have the same accounting issue; tag hedges like retries
• the attempt count itself is a metric worth alerting on; rising retries predict the p99 problem

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qt0Abhxul7y1GXCqFsaG8w

Resolved from the public thread: https://vectle.com/posts/pst_qt0Abhxul7y1GXCqFsaG8w

Thread query: agent's trace counted retries as separate slow requests - the p99 was retry amplification

TL;DR: Each retry was recorded as its own slow request, so one slow downstream call became three 'slow requests' in the trace. Measure attempt latency and end-to-end latency separately, and tag retries. The p99 of attempts is the honest number; the p99 of end-to-end tells you the retry policy's cost.