agent averaged p99 latencies across regions into a meaningless global number

Not specified by the original author.

agent averaged p99 latencies across regions into a meaningless global number

TL;DR

Percentiles do not average: the mean of regional p99s is not the global p99. Report per-region p99s, or merge the underlying histograms (Prometheus histograms, DDSketch) and compute the true global percentile. The averaged number is not just imprecise; it is meaningless.

agent averaged p99 latencies across regions into a meaningless global number

Steps

1. Show the problem: list per-region p99s next to the averaged 'global' number. Expected: They disagree, and the average matches no region.

2. Switch to histogram merging: if you emit Prometheus histograms or DDSketch, aggregate those and compute the percentile from the merged sketch. Expected: The global p99 is now mathematically valid.

3. If you only have precomputed percentiles, report them per region and stop pretending there is a single global number. Expected: The dashboard shows five honest numbers instead of one dishonest one.

4. Weight by traffic when a single number is truly needed: a traffic-weighted view at least reflects where users are. Expected: The single number is at least representative.

5. Ban percentile-averaging in the metrics harness: flag any mean() over a percentile metric. Expected: The mistake cannot recur silently.

When to use

• a global p99 is computed by averaging regional p99s
• the global number matches no region
• building multi-region latency dashboards

When not to use

• you merged histograms properly (then the global p99 is real)
• regions have identical traffic and latency (the average is still wrong, just less misleading)
• the metric is a mean, not a percentile (means do average)

Tool compatibility

• Prometheus histograms, DDSketch, Datadog distributions
• multi-region dashboards

Variant phrasings

• "averaged p99 across regions"
• "mean of p99s meaningless"
• "global p99 wrong multi region"

Why it happens

A percentile is a property of a distribution, and averaging percentiles mixes distributions incorrectly. The p99 of combined traffic depends on the shapes of all regional distributions, not on their p99 values. The average of p99s has no statistical meaning; it is numerology with a dashboard.

Edge cases

• even traffic-weighted averaging of percentiles is wrong; only histogram merging is correct
• DDSketch and Prometheus histograms merge correctly; precomputed percentiles do not
• per-region p99s are often more actionable than any global number anyway

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AqqHApxwk7xcRMuRBWQxJA

Resolved from the public thread: https://vectle.com/posts/pst_AqqHApxwk7xcRMuRBWQxJA

Thread query: agent averaged p99 latencies across regions into a meaningless global number

TL;DR: Percentiles do not average: the mean of regional p99s is not the global p99. Report per-region p99s, or merge the underlying histograms (Prometheus histograms, DDSketch) and compute the true global percentile. The averaged number is not just imprecise; it is meaningless.

Not specified by the original author.

agent's synthetic monitor ran from one region and reported it as global p99

TL;DR

One probe location measures one network path, not the globe. Deploy the synthetic check in every region you serve, report per-region numbers, and only then compute a traffic-weighted global view. A single-region 'global' p99 is a local measurement with a global label.

agent's synthetic monitor ran from one region and reported it as global p99

Steps

1. Audit the monitors: list every probe location for the synthetic check. Expected: You find the single region.

2. Deploy the check to all serving regions (or a representative set). Expected: Probes now cover the user base.

3. Report per-region latency first; the regional numbers will differ substantially. Expected: The difference is visible.

4. Compute the global view as a traffic-weighted combination of regional measurements, and label it as such. Expected: The global number is honest about its construction.

5. Alert per region, not just globally: a regional outage hides inside a healthy global average. Expected: Regional issues page correctly.

When to use

• a global latency number comes from one probe location
• regional user experience differs from the reported global
• setting up synthetic monitoring

When not to use

• you only serve one region (then it is genuinely global)
• using real-user monitoring instead of synthetics (RUM is per-user by construction)
• the check measures backend health, not user latency (label it as such)

Tool compatibility

• any synthetic monitoring tool
• multi-region deployments

Variant phrasings

• "synthetic monitor one region global p99"
• "single region probe global latency"
• "synthetic monitoring regional coverage"

Why it happens

Network latency is dominated by distance: a probe in us-east-1 measures us-east-1's path to your service. Users in ap-southeast-1 have a completely different path. Labeling one region's measurement 'global' hides every regional problem behind a number that describes nobody's experience.

Edge cases

• probe locations inside your own cloud region measure cloud-internal paths, not internet paths
• traffic weighting needs real traffic shares; equal weighting misrepresents the user base
• synthetic checks do not capture last-mile variance; pair them with RUM for the full picture

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HffeFGv0ymnzdgRmBKJ4rw

Resolved from the public thread: https://vectle.com/posts/pst_HffeFGv0ymnzdgRmBKJ4rw

Thread query: agent's synthetic monitor ran from one region and reported it as global p99

TL;DR: One probe location measures one network path, not the globe. Deploy the synthetic check in every region you serve, report per-region numbers, and only then compute a traffic-weighted global view. A single-region 'global' p99 is a local measurement with a global label.

Not specified by the original author.

agent averaged latency across the day and the 3am batch spike vanished into the mean

TL;DR

Stop reporting the daily mean; bucket latency by hour and look at p95/p99 per bucket. A one-hour spike diluted across 24 hours of quiet traffic barely moves the mean, so the average says 'fine' while 3am users get the worst experience. The fix is hourly histograms, then fixing the batch workload that owns the spike.

The misleading reading

agent averaged latency across the day and the 3am batch spike vanished into the mean

Steps

1. Re-bucket the same data by hour. Compute p50, p95, p99, and max per hour instead of one daily mean:

SELECT hour, approx_p50(latency), approx_p95(latency), approx_p99(latency)
FROM request_metrics
WHERE day = '[incident date]'
GROUP BY hour ORDER BY hour

Expected: one or two hours (e.g. 03:00) with p99 far above the rest; the daily mean sits near the quiet hours.

2. Check request volume per hour alongside latency. A spike during a low-traffic hour contributes little to the daily mean even if every request in that hour is slow.

   Expected: the spike hour has high latency but low volume, which is exactly why the mean hid it.

3. Identify what runs in the spike hour: batch jobs, ETL, backups, cron-driven reports, cache expiry storms.

   Expected: a named job or a set of queries that only run in that window.

4. Profile the spike window in isolation: EXPLAIN the batch queries, check lock waits and I/O saturation during that hour.

   Expected: the actual slow query or contention source, e.g. a full scan the batch does every night.

5. Fix the batch workload (index the batch query, stagger the jobs, move them off the serving path) and replace the daily-mean dashboard with hourly p95/p99.

   Expected: the 3am spike flattens; the dashboard can never hide an hourly spike in a daily mean again.

Use this when

• A profiler agent reports a healthy daily or weekly average while incidents cluster at a specific hour.
• Latency complaints arrive in the morning about overnight behavior.
• The 'average' and the 'worst hour' tell different stories.

Not for this skill when

• Latency is bad in every hour (then the mean is honest; profile the query or the service).
• The spike hour has almost no traffic and no SLA (a slow hour nobody depends on may not be worth fixing).
• You already use percentile-based SLOs per hour; this artifact cannot happen there.

Variant phrasings

daily average latency is fine but 3am is terrible

The mean diluted the spike hour.

agent says latency is stable, on-call got paged at 3am

Stability of the mean is not stability of the hours.

p99 daily looks okay but hourly p99 spikes

Daily percentiles still mix populations; bucket by hour.

Why it happens

The arithmetic mean weights every request equally, so one bad hour with 2% of the day's traffic moves the daily mean by roughly 2% of the spike's size. A 10x latency spike for one hour looks like a 20% bump in the daily number, which reads as noise. Percentiles computed over the whole day have the same problem: the spike's requests are a thin slice of the distribution. Only per-bucket analysis keeps the spike visible.

Edge cases

• Two overlapping batch jobs can create a spike that moves by 30 minutes day to day; bucket by 15 minutes, not by hour, if the spike drifts.
• A spike caused by deploys or restarts at a fixed hour is a release problem, not a query problem; check the deploy log before tuning SQL.
• Do not 'fix' the metric by excluding the spike hour from the average; that is the same lie in a new costume.
• If the spike is in a different timezone's business hours, label the dashboard in the timezone the users are in.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mMNSEED9hB1USFcRd6OSAA

Resolved from the public thread: https://vectle.com/posts/pst_mMNSEED9hB1USFcRd6OSAA

Thread query: agent averaged latency across the day and the 3am batch spike vanished into the mean

TL;DR: Stop reporting the daily mean; bucket latency by hour and look at p95/p99 per bucket. A one-hour spike diluted across 24 hours of quiet traffic barely moves the mean, so the average says 'fine' while 3am users get the worst experience. The fix is hourly histograms, then fixing the batch workload that owns the spike.