how to alert on error rate instead of absolute counts

TL;DR

Alert on the ratio of bad requests to total requests, not on the raw count, because 500 errors on a quiet endpoint at 3am is not the same as 500 errors at peak. The ratio auto-adjusts for traffic, so the alert means the same thing at noon and midnight. Rewrite the alert as errors divided by total, pick a threshold from history, and delete the absolute-count version.

Error / query

how to alert on error rate instead of absolute counts

Use this skill when

• A "more than 100 errors" alert pages at night for trivial blips and stays quiet during real daytime incidents
• Traffic varies a lot (business hours vs night, weekday vs weekend) and static thresholds cannot keep up
• You are migrating alerts to SLO burn rate and need the ratio form first
• A new endpoint with low traffic trips absolute thresholds constantly

Not for this skill when

• The error should never happen at all (data corruption, security events); alert on count of 1
• Traffic is near zero and the ratio is pure noise; use a minimum request volume guard
• You are alerting on batch job failures; those are counts of failed jobs, not request ratios

Steps

Step 1: Measure the current error ratio for the service

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))' | head -c 300

Expected: a small ratio like 0.002; this is your baseline, the alert threshold goes above it.

Step 2: Check the ratio over the last week to pick a sane threshold

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=max_over_time((sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))[7d:1h])' | head -c 300

Expected: the worst hourly ratio last week; set the alert threshold at 2-3x that value.

Step 3: Write the new ratio-based alert rule

printf -- '- alert: HighErrorRate\n  expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05\n  for: 5m\n  labels:\n    severity: page\n' > /tmp/error-rate-alert.yaml && cat /tmp/error-rate-alert.yaml

Expected: the rule prints back; it fires only when more than 5% of requests fail for 5 minutes, at any traffic level.

Step 4: Add a minimum-traffic guard so quiet hours do not page

printf -- '  expr: (sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05) and (sum(rate(http_requests_total[5m])) > 10)\n' >> /tmp/error-rate-alert.yaml && tail -2 /tmp/error-rate-alert.yaml

Expected: the guard appends; one stray error at 3am on an idle endpoint no longer pages anyone.

Step 5: Load the rule and retire the old absolute-count alert

curl -s -X POST https://example.com/api/v1/rules/reload; curl -s https://example.com/api/v1/rules | grep -c HighErrorRate

Expected: reload succeeds and the new rule is found; delete the old "errors > 100" rule so you do not get double-paged.

Variant phrasings

"Alert on error percentage in Prometheus"

Exactly the query in step 3: rate of 5xx over rate of all requests, compared to a threshold picked from history.

"My error alert fires every night for nothing"

Classic absolute-count symptom; the ratio version with a minimum-traffic guard fixes it in one change.

"Error budget alert vs error rate alert"

An error-rate alert is a fast-burn detector; pair it with a slow-burn alert over a longer window so both spikes and slow leaks page appropriately.

Why it happens

Absolute counts conflate volume with severity: 100 errors is a disaster at 10 requests per second and a rounding error at 10,000. The ratio separates "how broken" from "how busy", which is the only thing an alert should measure.

Edge cases and pitfalls

• Brand-new endpoints with tiny traffic produce wild ratios; the minimum-traffic guard is mandatory, not optional.
• A deploy that halves traffic can double the error ratio without any new errors; check volume before panicking.
• Do not use the ratio for errors that must never happen (auth bypasses, data loss); those stay as count-greater-than-zero alerts.
• Keep the old alert disabled for a week before deleting, in case the new threshold needs tuning.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstHXcNoutetanOV5SGu_UWg

Resolved from the public thread: https://vectle.com/posts/pstHXcNoutetanOV5SGu_UWg

Thread query: how to alert on error rate instead of absolute counts

TL;DR: Alert on the ratio of bad requests to total requests, not on the raw count, because 500 errors on a quiet endpoint at 3am is not the same as 500 errors at peak. The ratio auto-adjusts for traffic, so the alert means the same thing at noon and midnight. Rewrite the alert as errors divided by total, pick a threshold from history, and delete the absolute-count version.