## TL;DR
The monitor alerts on the first bad reading, so every transient null becomes a page. Require the breach to persist (N consecutive failures or a sustained window), add a per-metric cooldown after each alert, and treat null readings as "unknown" rather than "breaching". Real outages still page within minutes; blips stay silent.

## Symptom
The monitoring agent sends a burst of alerts for null or out-of-range readings that resolve on their own within a minute. On-call learns to ignore the channel, which is exactly when a real incident gets missed. Alert volume is high, signal is near zero.

## Steps

1. Require the breach to persist: alert only after N consecutive failing evaluations or a sustained window (for example 3 checks in a row, or 5 minutes breached), not on a single bad reading.
Expected output: isolated nulls produce no alert; a genuinely stuck metric still fires within minutes.

2. Add a per-metric cooldown: after one alert, suppress repeats for the same metric until it recovers or the cooldown expires.
Expected output: one incident produces one alert thread, not fifty.

3. Treat null or missing readings as "unknown", not "breaching". A separate, quieter path can track data freshness; the paging path should only fire on confirmed bad values.
Expected output: transient ingestion gaps stop paging anyone.

4. Test with a synthetic blip: inject one null reading and confirm silence, then inject a sustained breach and confirm exactly one alert.
Expected output: both behaviors verified before the change ships.

## When to use
- A monitor pages on single bad or null readings
- Alert fatigue is setting in on a channel the agent writes to
- The monitored metric is naturally spiky or has ingestion gaps

## When not to use
- The metric is a hard SLO burn signal (those want fast alerts by design; tune the threshold, not the debounce)
- Alerts are already rare and every one was real (do not add latency to a working signal)
- The problem is alert routing, not alert frequency

## Variant phrasings

### alerts fire during deploys then resolve
Deploy-time blips are transient by nature. Either debounce as above, or add a deploy-aware quiet period; do not just raise thresholds.

### one bad sensor pages the whole team nightly
Same pattern at small scale. Consecutive-breach requirements kill it without losing real failures.

## Why it happens
Monitors are usually built and tested against steady-state data, where any deviation looks meaningful. In production, nulls from ingestion hiccups, restarts, and deploy windows are routine. A monitor with no persistence requirement cannot tell "one bad sample" from "the system is down", so it pages for both.

## Edge cases
- A real outage that starts during a cooldown still needs escalation; use severity escalation (page again if the breach persists past a longer window) rather than a hard mute.
- Flapping (breach, recover, breach) can evade consecutive-count logic; a sustained-window rule catches flapping better than a count.
- Do not debounce the "data is stale" signal into silence too; freshness deserves its own quiet alert, separate from the value alert.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CjjQHPCTaKiEdRhX4JhdcQ