agent averaged p99 latencies across regions into a meaningless global number

agent averaged p99 latencies across regions into a meaningless global number

TL;DR

Percentiles do not average: the mean of regional p99s is not the global p99. Report per-region p99s, or merge the underlying histograms (Prometheus histograms, DDSketch) and compute the true global percentile. The averaged number is not just imprecise; it is meaningless.

agent averaged p99 latencies across regions into a meaningless global number

Steps

1. Show the problem: list per-region p99s next to the averaged 'global' number. Expected: They disagree, and the average matches no region.

2. Switch to histogram merging: if you emit Prometheus histograms or DDSketch, aggregate those and compute the percentile from the merged sketch. Expected: The global p99 is now mathematically valid.

3. If you only have precomputed percentiles, report them per region and stop pretending there is a single global number. Expected: The dashboard shows five honest numbers instead of one dishonest one.

4. Weight by traffic when a single number is truly needed: a traffic-weighted view at least reflects where users are. Expected: The single number is at least representative.

5. Ban percentile-averaging in the metrics harness: flag any mean() over a percentile metric. Expected: The mistake cannot recur silently.

When to use

• a global p99 is computed by averaging regional p99s
• the global number matches no region
• building multi-region latency dashboards

When not to use

• you merged histograms properly (then the global p99 is real)
• regions have identical traffic and latency (the average is still wrong, just less misleading)
• the metric is a mean, not a percentile (means do average)

Tool compatibility

• Prometheus histograms, DDSketch, Datadog distributions
• multi-region dashboards

Variant phrasings

• "averaged p99 across regions"
• "mean of p99s meaningless"
• "global p99 wrong multi region"

Why it happens

A percentile is a property of a distribution, and averaging percentiles mixes distributions incorrectly. The p99 of combined traffic depends on the shapes of all regional distributions, not on their p99 values. The average of p99s has no statistical meaning; it is numerology with a dashboard.

Edge cases

• even traffic-weighted averaging of percentiles is wrong; only histogram merging is correct
• DDSketch and Prometheus histograms merge correctly; precomputed percentiles do not
• per-region p99s are often more actionable than any global number anyway

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AqqHApxwk7xcRMuRBWQxJA