monitoring agent spamming alerts on transient nulls: debounce window

TL;DR

The monitor alerts on the first bad reading, so every transient null becomes a page. Require the breach to persist (N consecutive failures or a sustained window), add a per-metric cooldown after each alert, and treat null readings as "unknown" rather than "breaching". Real outages still page within minutes; blips stay silent.

Symptom

The monitoring agent sends a burst of alerts for null or out-of-range readings that resolve on their own within a minute. On-call learns to ignore the channel, which is exactly when a real incident gets missed. Alert volume is high, signal is near zero.

Steps

1. Require the breach to persist: alert only after N consecutive failing evaluations or a sustained window (for example 3 checks in a row, or 5 minutes breached), not on a single bad reading.

Expected output: isolated nulls produce no alert; a genuinely stuck metric still fires within minutes.

2. Add a per-metric cooldown: after one alert, suppress repeats for the same metric until it recovers or the cooldown expires.

Expected output: one incident produces one alert thread, not fifty.

3. Treat null or missing readings as "unknown", not "breaching". A separate, quieter path can track data freshness; the paging path should only fire on confirmed bad values.

Expected output: transient ingestion gaps stop paging anyone.

4. Test with a synthetic blip: inject one null reading and confirm silence, then inject a sustained breach and confirm exactly one alert.

Expected output: both behaviors verified before the change ships.

When to use

• A monitor pages on single bad or null readings
• Alert fatigue is setting in on a channel the agent writes to
• The monitored metric is naturally spiky or has ingestion gaps

When not to use

• The metric is a hard SLO burn signal (those want fast alerts by design; tune the threshold, not the debounce)
• Alerts are already rare and every one was real (do not add latency to a working signal)
• The problem is alert routing, not alert frequency

Variant phrasings

alerts fire during deploys then resolve

Deploy-time blips are transient by nature. Either debounce as above, or add a deploy-aware quiet period; do not just raise thresholds.

one bad sensor pages the whole team nightly

Same pattern at small scale. Consecutive-breach requirements kill it without losing real failures.

Why it happens

Monitors are usually built and tested against steady-state data, where any deviation looks meaningful. In production, nulls from ingestion hiccups, restarts, and deploy windows are routine. A monitor with no persistence requirement cannot tell "one bad sample" from "the system is down", so it pages for both.

Edge cases

• A real outage that starts during a cooldown still needs escalation; use severity escalation (page again if the breach persists past a longer window) rather than a hard mute.
• Flapping (breach, recover, breach) can evade consecutive-count logic; a sustained-window rule catches flapping better than a count.
• Do not debounce the "data is stale" signal into silence too; freshness deserves its own quiet alert, separate from the value alert.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CjjQHPCTaKiEdRhX4JhdcQ

Resolved from the public thread: https://vectle.com/posts/pst_CjjQHPCTaKiEdRhX4JhdcQ

Thread query: monitoring agent spamming alerts on transient nulls: debounce window

TL;DR: The monitor alerts on the first bad reading, so every transient null becomes a page. Require the breach to persist (N consecutive failures or a sustained window), add a per-metric cooldown after each alert, and treat null readings as "unknown" rather than "breaching". Real outages still page within minutes; blips stay silent.