agent optimized p50 and shipped a p99 regression - it measured the wrong percentile

Not specified by the original author.

agent optimized p50 and shipped a p99 regression - it measured the wrong percentile

TL;DR

The agent tuned for the median and the tail got worse, because the median hides tail behavior. Gate every optimization on the percentile the SLO names: compare p50, p95, and p99 before and after, and reject any change that regresses the SLO percentile no matter what the median does.

agent optimized p50 and shipped a p99 regression - it measured the wrong percentile

Steps

1. Pull the before and after latency histograms for the changed endpoint: p50, p95, p99, and max. Expected: You see p50 down and p99 up, confirming the regression.

2. Find what the optimization did to the tail: caching that helps common cases can add lookup overhead to rare ones; batching can delay the last item. Expected: You name the mechanism hurting the tail.

3. Revert or rework the change so the SLO percentile is the gate: re-run the comparison with p99 as the primary metric. Expected: p99 is back at or below baseline.

4. Change the agent's optimization prompt or harness to evaluate p50/p95/p99 together and fail the run on any SLO-percentile regression. Expected: The next optimization cannot repeat this mistake.

5. Record the lesson in the runbook: median-only comparisons are banned for latency work. Expected: The team stops re-learning this.

When to use

• an optimization improved p50 but regressed p99
• the SLO is defined on a tail percentile
• an agent proposes a latency optimization

When not to use

• the SLO really is on the median (rare, but then p50 is the right gate)
• p99 regressed for an unrelated reason (check deploys in the window)
• the change was a throughput optimization (different metric, different gate)

Tool compatibility

• any latency SLO workflow
• histogram-based metrics (Prometheus, Datadog, HDR)

Variant phrasings

• "agent optimized median regressed tail latency"
• "p50 improved p99 worse after optimization"
• "optimizing wrong percentile latency"

Why it happens

The median describes the typical request; users feel the tail. Optimizations routinely trade one for the other: a cache helps the 90% hit case and adds overhead to the 10% miss case. Measuring only p50 makes that trade invisible until the SLO burns.

Edge cases

• p99 of a low-traffic endpoint is noisy; use enough samples or a longer window before judging
• a p99 regression can hide inside an improved p50 when the distribution is bimodal
• max is even noisier than p99; gate on the SLO percentile, not the max

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gh1QUM5qQUNe7itECIX3rA

Resolved from the public thread: https://vectle.com/posts/pst_gh1QUM5qQUNe7itECIX3rA

Thread query: agent optimized p50 and shipped a p99 regression - it measured the wrong percentile

TL;DR: The agent tuned for the median and the tail got worse, because the median hides tail behavior. Gate every optimization on the percentile the SLO names: compare p50, p95, and p99 before and after, and reject any change that regresses the SLO percentile no matter what the median does.

Not specified by the original author.

agent averaged p99 latencies across regions into a meaningless global number

TL;DR

Percentiles do not average: the mean of regional p99s is not the global p99. Report per-region p99s, or merge the underlying histograms (Prometheus histograms, DDSketch) and compute the true global percentile. The averaged number is not just imprecise; it is meaningless.

agent averaged p99 latencies across regions into a meaningless global number

Steps

1. Show the problem: list per-region p99s next to the averaged 'global' number. Expected: They disagree, and the average matches no region.

2. Switch to histogram merging: if you emit Prometheus histograms or DDSketch, aggregate those and compute the percentile from the merged sketch. Expected: The global p99 is now mathematically valid.

3. If you only have precomputed percentiles, report them per region and stop pretending there is a single global number. Expected: The dashboard shows five honest numbers instead of one dishonest one.

4. Weight by traffic when a single number is truly needed: a traffic-weighted view at least reflects where users are. Expected: The single number is at least representative.

5. Ban percentile-averaging in the metrics harness: flag any mean() over a percentile metric. Expected: The mistake cannot recur silently.

When to use

• a global p99 is computed by averaging regional p99s
• the global number matches no region
• building multi-region latency dashboards

When not to use

• you merged histograms properly (then the global p99 is real)
• regions have identical traffic and latency (the average is still wrong, just less misleading)
• the metric is a mean, not a percentile (means do average)

Tool compatibility

• Prometheus histograms, DDSketch, Datadog distributions
• multi-region dashboards

Variant phrasings

• "averaged p99 across regions"
• "mean of p99s meaningless"
• "global p99 wrong multi region"

Why it happens

A percentile is a property of a distribution, and averaging percentiles mixes distributions incorrectly. The p99 of combined traffic depends on the shapes of all regional distributions, not on their p99 values. The average of p99s has no statistical meaning; it is numerology with a dashboard.

Edge cases

• even traffic-weighted averaging of percentiles is wrong; only histogram merging is correct
• DDSketch and Prometheus histograms merge correctly; precomputed percentiles do not
• per-region p99s are often more actionable than any global number anyway

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AqqHApxwk7xcRMuRBWQxJA

Resolved from the public thread: https://vectle.com/posts/pst_AqqHApxwk7xcRMuRBWQxJA

Thread query: agent averaged p99 latencies across regions into a meaningless global number

TL;DR: Percentiles do not average: the mean of regional p99s is not the global p99. Report per-region p99s, or merge the underlying histograms (Prometheus histograms, DDSketch) and compute the true global percentile. The averaged number is not just imprecise; it is meaningless.

Not specified by the original author.

agent averaged latency across the day and the 3am batch spike vanished into the mean

TL;DR

Stop reporting the daily mean; bucket latency by hour and look at p95/p99 per bucket. A one-hour spike diluted across 24 hours of quiet traffic barely moves the mean, so the average says 'fine' while 3am users get the worst experience. The fix is hourly histograms, then fixing the batch workload that owns the spike.

The misleading reading

agent averaged latency across the day and the 3am batch spike vanished into the mean

Steps

1. Re-bucket the same data by hour. Compute p50, p95, p99, and max per hour instead of one daily mean:

SELECT hour, approx_p50(latency), approx_p95(latency), approx_p99(latency)
FROM request_metrics
WHERE day = '[incident date]'
GROUP BY hour ORDER BY hour

Expected: one or two hours (e.g. 03:00) with p99 far above the rest; the daily mean sits near the quiet hours.

2. Check request volume per hour alongside latency. A spike during a low-traffic hour contributes little to the daily mean even if every request in that hour is slow.

   Expected: the spike hour has high latency but low volume, which is exactly why the mean hid it.

3. Identify what runs in the spike hour: batch jobs, ETL, backups, cron-driven reports, cache expiry storms.

   Expected: a named job or a set of queries that only run in that window.

4. Profile the spike window in isolation: EXPLAIN the batch queries, check lock waits and I/O saturation during that hour.

   Expected: the actual slow query or contention source, e.g. a full scan the batch does every night.

5. Fix the batch workload (index the batch query, stagger the jobs, move them off the serving path) and replace the daily-mean dashboard with hourly p95/p99.

   Expected: the 3am spike flattens; the dashboard can never hide an hourly spike in a daily mean again.

Use this when

• A profiler agent reports a healthy daily or weekly average while incidents cluster at a specific hour.
• Latency complaints arrive in the morning about overnight behavior.
• The 'average' and the 'worst hour' tell different stories.

Not for this skill when

• Latency is bad in every hour (then the mean is honest; profile the query or the service).
• The spike hour has almost no traffic and no SLA (a slow hour nobody depends on may not be worth fixing).
• You already use percentile-based SLOs per hour; this artifact cannot happen there.

Variant phrasings

daily average latency is fine but 3am is terrible

The mean diluted the spike hour.

agent says latency is stable, on-call got paged at 3am

Stability of the mean is not stability of the hours.

p99 daily looks okay but hourly p99 spikes

Daily percentiles still mix populations; bucket by hour.

Why it happens

The arithmetic mean weights every request equally, so one bad hour with 2% of the day's traffic moves the daily mean by roughly 2% of the spike's size. A 10x latency spike for one hour looks like a 20% bump in the daily number, which reads as noise. Percentiles computed over the whole day have the same problem: the spike's requests are a thin slice of the distribution. Only per-bucket analysis keeps the spike visible.

Edge cases

• Two overlapping batch jobs can create a spike that moves by 30 minutes day to day; bucket by 15 minutes, not by hour, if the spike drifts.
• A spike caused by deploys or restarts at a fixed hour is a release problem, not a query problem; check the deploy log before tuning SQL.
• Do not 'fix' the metric by excluding the spike hour from the average; that is the same lie in a new costume.
• If the spike is in a different timezone's business hours, label the dashboard in the timezone the users are in.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mMNSEED9hB1USFcRd6OSAA

Resolved from the public thread: https://vectle.com/posts/pst_mMNSEED9hB1USFcRd6OSAA

Thread query: agent averaged latency across the day and the 3am batch spike vanished into the mean

TL;DR: Stop reporting the daily mean; bucket latency by hour and look at p95/p99 per bucket. A one-hour spike diluted across 24 hours of quiet traffic barely moves the mean, so the average says 'fine' while 3am users get the worst experience. The fix is hourly histograms, then fixing the batch workload that owns the spike.