Diag: burn-rate alert misfiring, check window math before thresholds

Diag: SLO burn-rate alert misfiring

Symptom (exact): burn-rate alert pages, but the service looks healthy and the error budget looks intact in the SLO view.

Likely causes: (a) alert window too short for the traffic pattern (low-traffic service, one bad minute burns the fast window), (b) SLO query changed (new tags split the good/bad events differently), (c) the underlying monitor feeding a monitor-based SLO flaps, (d) time zone / window alignment confusion.

Confirm:

1. Open the SLO: what is the actual budget remaining? If it is healthy, the alert math disagrees with the SLO math, which points at (a) or (b).
2. Look at the alert evaluation: which window burned? A 1h window burning on a service with 10 requests/hour is noise, not signal.
3. Diff the SLO query against last week. A tag rename or a new status code in the bad-events filter changes the ratio without changing reality.
4. For monitor-based SLOs: is the underlying monitor flapping on no-data? Fix the monitor first.

Fix: lengthen the fast-burn window for low-traffic services, fix the query, or split the SLO per traffic class. Do not just raise the threshold: that hides the next real breach.

Verify: the alert stays quiet through a normal traffic cycle and fires on a synthetic error burst (test with a canary if you can). An alert you have never seen fire correctly is a decoration.

Datadog SLOs support time-slice and monitor-based types with burn-rate alerting; a misfiring burn alert is usually window math, not a real budget breach. Verified against live docs 2026-09-26.