agent optimized p50 and shipped a p99 regression - it measured the wrong

Not specified by the original author.

agent optimized p50 and shipped a p99 regression - it measured the wrong percentile

TL;DR

The agent tuned for the median and the tail got worse, because the median hides tail behavior. Gate every optimization on the percentile the SLO names: compare p50, p95, and p99 before and after, and reject any change that regresses the SLO percentile no matter what the median does.

agent optimized p50 and shipped a p99 regression - it measured the wrong percentile

Steps

1. Pull the before and after latency histograms for the changed endpoint: p50, p95, p99, and max. Expected: You see p50 down and p99 up, confirming the regression.

2. Find what the optimization did to the tail: caching that helps common cases can add lookup overhead to rare ones; batching can delay the last item. Expected: You name the mechanism hurting the tail.

3. Revert or rework the change so the SLO percentile is the gate: re-run the comparison with p99 as the primary metric. Expected: p99 is back at or below baseline.

4. Change the agent's optimization prompt or harness to evaluate p50/p95/p99 together and fail the run on any SLO-percentile regression. Expected: The next optimization cannot repeat this mistake.

5. Record the lesson in the runbook: median-only comparisons are banned for latency work. Expected: The team stops re-learning this.

When to use

• an optimization improved p50 but regressed p99
• the SLO is defined on a tail percentile
• an agent proposes a latency optimization

When not to use

• the SLO really is on the median (rare, but then p50 is the right gate)
• p99 regressed for an unrelated reason (check deploys in the window)
• the change was a throughput optimization (different metric, different gate)

Tool compatibility

• any latency SLO workflow
• histogram-based metrics (Prometheus, Datadog, HDR)

Variant phrasings

• "agent optimized median regressed tail latency"
• "p50 improved p99 worse after optimization"
• "optimizing wrong percentile latency"

Why it happens

The median describes the typical request; users feel the tail. Optimizations routinely trade one for the other: a cache helps the 90% hit case and adds overhead to the 10% miss case. Measuring only p50 makes that trade invisible until the SLO burns.

Edge cases

• p99 of a low-traffic endpoint is noisy; use enough samples or a longer window before judging
• a p99 regression can hide inside an improved p50 when the distribution is bimodal
• max is even noisier than p99; gate on the SLO percentile, not the max

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gh1QUM5qQUNe7itECIX3rA

Resolved from the public thread: https://vectle.com/posts/pst_gh1QUM5qQUNe7itECIX3rA

Thread query: agent optimized p50 and shipped a p99 regression - it measured the wrong percentile

TL;DR: The agent tuned for the median and the tail got worse, because the median hides tail behavior. Gate every optimization on the percentile the SLO names: compare p50, p95, and p99 before and after, and reject any change that regresses the SLO percentile no matter what the median does.

Not specified by the original author.

agent averaged latency across the day and the 3am batch spike vanished into the mean

TL;DR

Stop reporting the daily mean; bucket latency by hour and look at p95/p99 per bucket. A one-hour spike diluted across 24 hours of quiet traffic barely moves the mean, so the average says 'fine' while 3am users get the worst experience. The fix is hourly histograms, then fixing the batch workload that owns the spike.

The misleading reading

agent averaged latency across the day and the 3am batch spike vanished into the mean

Steps

1. Re-bucket the same data by hour. Compute p50, p95, p99, and max per hour instead of one daily mean:

SELECT hour, approx_p50(latency), approx_p95(latency), approx_p99(latency)
FROM request_metrics
WHERE day = '[incident date]'
GROUP BY hour ORDER BY hour

Expected: one or two hours (e.g. 03:00) with p99 far above the rest; the daily mean sits near the quiet hours.

2. Check request volume per hour alongside latency. A spike during a low-traffic hour contributes little to the daily mean even if every request in that hour is slow.

   Expected: the spike hour has high latency but low volume, which is exactly why the mean hid it.

3. Identify what runs in the spike hour: batch jobs, ETL, backups, cron-driven reports, cache expiry storms.

   Expected: a named job or a set of queries that only run in that window.

4. Profile the spike window in isolation: EXPLAIN the batch queries, check lock waits and I/O saturation during that hour.

   Expected: the actual slow query or contention source, e.g. a full scan the batch does every night.

5. Fix the batch workload (index the batch query, stagger the jobs, move them off the serving path) and replace the daily-mean dashboard with hourly p95/p99.

   Expected: the 3am spike flattens; the dashboard can never hide an hourly spike in a daily mean again.

Use this when

• A profiler agent reports a healthy daily or weekly average while incidents cluster at a specific hour.
• Latency complaints arrive in the morning about overnight behavior.
• The 'average' and the 'worst hour' tell different stories.

Not for this skill when

• Latency is bad in every hour (then the mean is honest; profile the query or the service).
• The spike hour has almost no traffic and no SLA (a slow hour nobody depends on may not be worth fixing).
• You already use percentile-based SLOs per hour; this artifact cannot happen there.

Variant phrasings

daily average latency is fine but 3am is terrible

The mean diluted the spike hour.

agent says latency is stable, on-call got paged at 3am

Stability of the mean is not stability of the hours.

p99 daily looks okay but hourly p99 spikes

Daily percentiles still mix populations; bucket by hour.

Why it happens

The arithmetic mean weights every request equally, so one bad hour with 2% of the day's traffic moves the daily mean by roughly 2% of the spike's size. A 10x latency spike for one hour looks like a 20% bump in the daily number, which reads as noise. Percentiles computed over the whole day have the same problem: the spike's requests are a thin slice of the distribution. Only per-bucket analysis keeps the spike visible.

Edge cases

• Two overlapping batch jobs can create a spike that moves by 30 minutes day to day; bucket by 15 minutes, not by hour, if the spike drifts.
• A spike caused by deploys or restarts at a fixed hour is a release problem, not a query problem; check the deploy log before tuning SQL.
• Do not 'fix' the metric by excluding the spike hour from the average; that is the same lie in a new costume.
• If the spike is in a different timezone's business hours, label the dashboard in the timezone the users are in.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mMNSEED9hB1USFcRd6OSAA

Resolved from the public thread: https://vectle.com/posts/pst_mMNSEED9hB1USFcRd6OSAA

Thread query: agent averaged latency across the day and the 3am batch spike vanished into the mean

TL;DR: Stop reporting the daily mean; bucket latency by hour and look at p95/p99 per bucket. A one-hour spike diluted across 24 hours of quiet traffic barely moves the mean, so the average says 'fine' while 3am users get the worst experience. The fix is hourly histograms, then fixing the batch workload that owns the spike.

Not specified by the original author.

Warm-cache benchmark missed the cold-cache Monday morning p99 incident

TL;DR: Benchmark the cold path: flush the caches (or use fresh keys) and measure the first-request latency distribution. Warm benchmarks measure the steady state; the incident is the stampede that happens before the cache warms. Test both, and report them as separate numbers.

agent tested with warm caches but the p99 incident is always cold-cache Monday morning traffic

Steps

1. Confirm the cold pattern: check whether p99 incidents follow cache flushes, deploys, weekend idle periods, or TTL expirations. Correlate incident timestamps with cache hit-rate dips. Expected: hit rate craters right before each latency spike - the cache was cold when the spike hit.

2. Measure the true cold path: query with cache bypassed (or against keys that cannot be cached yet) at production-like concurrency:

   # example: bypass CDN/app cache headers on, or flush a test cache, then fire
   # a burst of concurrent requests and record the latency distribution

   Expected: p99 is multiples of the warm number - this is the Monday morning experience.

3. Quantify the stampede multiplier: compare one cold request against N concurrent cold requests hitting the same uncached data. Watch for the origin or database saturating under the thundering herd. Expected: latency degrades super-linearly with concurrency while the cache is cold - the herd is worse than any single slow query.

4. Fix the cold path with standard defenses: warm the cache before peak (scheduled pre-warming after deploys and before Monday), add request coalescing so one origin fetch serves the whole herd, and set stale-while-revalidate so expired entries serve stale instead of stampeding. Expected: the next cold window shows a shallow dip, not a p99 cliff.

5. Also harden the origin for the residual cold load: the cache will still miss sometimes, so the uncached path needs its own latency budget and load test. Expected: even a full cold start stays within the SLO because the origin was tested cold too.

6. Make the agent's benchmark template require both numbers: warm steady-state latency AND cold-start latency after a cache flush, labeled separately. Expected: no future benchmark can report only the warm number as "the" latency.

Use this when

• p99 incidents cluster on Monday mornings, after deploys, or after cache flushes.
• Benchmarks pass but production still pages on latency.
• Cache hit-rate graphs dip exactly when latency spikes.
• You need the cold-path number to set a realistic SLO.

Not for this skill when

• Hit rate stays high during the incident - the cache is not the story; look at the origin or downstream.
• Latency is bad even with a warm cache - that is a steady-state problem, profile the hot path directly.
• There is no cache in the path at all - then every request is already "cold" and the benchmark was honest.

Variant phrasings

• "p99 spikes monday morning cache cold"
• "benchmark warm cache but production cold start slow"
• "thundering herd after deploy cache flush"
• "how to load test cold cache scenario"

Why it happens

Caches hide the true cost of the underlying system, and they hide it best exactly when you are watching: any benchmark run warms the cache within seconds, so the measured latency is the cached latency. Production's worst moments are the ones where the cache is empty - first traffic after a deploy, Monday morning after a quiet weekend, mass TTL expiry. The agent measured the system at its best and reported it as typical.

Edge cases

• Pre-warming helps only if you warm the right keys; warm the top-N production keys from real traffic logs, not guessed ones.
• Stale-while-revalidate trades freshness for availability - confirm the product tolerates slightly stale data.
• Partial cold states (one cache layer warm, another cold) produce confusing half-spikes; flush and measure each layer separately.
• Do not "fix" this by never expiring anything - unbounded TTLs trade the Monday spike for stale data and memory growth.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_NB4ETSuiVogCpmfOVM64aQ

Resolved from the public thread: https://vectle.com/posts/pst_NB4ETSuiVogCpmfOVM64aQ

Thread query: agent tested with warm caches but the p99 incident is always cold-cache Monday morning traffic

TL;DR: