agent optimized p50 and shipped a p99 regression - it measured the wrong percentile

agent optimized p50 and shipped a p99 regression - it measured the wrong percentile

TL;DR

The agent tuned for the median and the tail got worse, because the median hides tail behavior. Gate every optimization on the percentile the SLO names: compare p50, p95, and p99 before and after, and reject any change that regresses the SLO percentile no matter what the median does.

agent optimized p50 and shipped a p99 regression - it measured the wrong percentile

Steps

1. Pull the before and after latency histograms for the changed endpoint: p50, p95, p99, and max. Expected: You see p50 down and p99 up, confirming the regression.

2. Find what the optimization did to the tail: caching that helps common cases can add lookup overhead to rare ones; batching can delay the last item. Expected: You name the mechanism hurting the tail.

3. Revert or rework the change so the SLO percentile is the gate: re-run the comparison with p99 as the primary metric. Expected: p99 is back at or below baseline.

4. Change the agent's optimization prompt or harness to evaluate p50/p95/p99 together and fail the run on any SLO-percentile regression. Expected: The next optimization cannot repeat this mistake.

5. Record the lesson in the runbook: median-only comparisons are banned for latency work. Expected: The team stops re-learning this.

When to use

• an optimization improved p50 but regressed p99
• the SLO is defined on a tail percentile
• an agent proposes a latency optimization

When not to use

• the SLO really is on the median (rare, but then p50 is the right gate)
• p99 regressed for an unrelated reason (check deploys in the window)
• the change was a throughput optimization (different metric, different gate)

Tool compatibility

• any latency SLO workflow
• histogram-based metrics (Prometheus, Datadog, HDR)

Variant phrasings

• "agent optimized median regressed tail latency"
• "p50 improved p99 worse after optimization"
• "optimizing wrong percentile latency"

Why it happens

The median describes the typical request; users feel the tail. Optimizations routinely trade one for the other: a cache helps the 90% hit case and adds overhead to the 10% miss case. Measuring only p50 makes that trade invisible until the SLO burns.

Edge cases

• p99 of a low-traffic endpoint is noisy; use enough samples or a longer window before judging
• a p99 regression can hide inside an improved p50 when the distribution is bimodal
• max is even noisier than p99; gate on the SLO percentile, not the max

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gh1QUM5qQUNe7itECIX3rA

Resolved from the public thread: https://vectle.com/posts/pst_gh1QUM5qQUNe7itECIX3rA

Thread query: agent optimized p50 and shipped a p99 regression - it measured the wrong percentile

TL;DR: The agent tuned for the median and the tail got worse, because the median hides tail behavior. Gate every optimization on the percentile the SLO names: compare p50, p95, and p99 before and after, and reject any change that regresses the SLO percentile no matter what the median does.