alert fatigue: how to tune noisy alerts

Not specified by the original author.

TL;DR

Kill or demote every alert that fires more often than it leads to action. The fix for fatigue is fewer, better alerts, not a better on-call attitude. An alert nobody acts on trains everyone to ignore all of them. Start by measuring which alerts fire without action, then delete or downgrade the top offenders.

Error / query

alert fatigue: how to tune noisy alerts

The pager goes off all night and nobody learns anything from it. People have started ignoring alerts, including the real ones.

Use this skill when

• on-call is drowning in pages nobody acts on
• alerts fire and resolve on their own repeatedly
• the team has started muting or ignoring the pager
• youre auditing an alert ruleset that grew without pruning

Not for this skill when

• you need to write new alerts from scratch (a different skill)
• the problem is missing alerts, not noisy ones (coverage gap, opposite fix)
• you need Alertmanager routing config (delivery mechanics, not tuning)

Steps

1. Find your noisiest alerts

Count alert firings over the last 7 days. The ones at the top of the list are your noise.

curl -s --get "http://YOUR-prometheus-host/api/v1/query_range" \
  --data-urlencode "query=ALERTS" \
  --data-urlencode "start=$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --data-urlencode "end=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --data-urlencode "step=1h" -o /tmp/alerts.json
python3 -c "
import json
d = json.load(open('/tmp/alerts.json'))
series = d['data']['result']
ranked = sorted(series, key [your value] s: len(s['values']), reverse=True)
for s in ranked[:10]:
    print(len(s['values']), s['metric'].get('alertname'), s['metric'].get('severity'))
"

Expected: the top 10 alerts by firing frequency. These are the candidates for demotion or deletion.

2. Demote or delete the top offenders

For each noisy alert, pick one: fix the threshold, demote page to ticket, or delete it. "Keep and ignore" is not an option.

grep -rn "severity: page" ./prometheus-rules/ | head -20

Expected: a list of paging alerts to review one by one. Each gets a fix, a demotion, or a deletion, decided this week, not someday.

3. Add a persistence window so transient blips dont page

Most flapping dies with a for clause. The alert should only page when the condition persists, not on one bad scrape.

cat > /tmp/alert-snippet.yaml <<'EOF'
# before: fires on any single scrape miss
# after: must persist 5 minutes before paging
- alert: ServiceDown
  expr: up == 0
  for: 5m
  labels:
    severity: page
EOF
cat /tmp/alert-snippet.yaml

Expected: alerts only page when the condition holds for the full window. Single-scrape blips stop waking people up.

Variant phrasings

how to reduce alert noise

Same fix: rank by noise (step 1), then demote, fix, or delete (step 2). Noise reduction is subtraction, not better thresholds alone.

too many prometheus alerts

Same fix: the method above is Prometheus-shaped but the principle is universal. Count firings, cut the ones with no action.

how to tune alerting thresholds

Same fix: tune only the alerts that survive step 2. Tuning an alert nobody acts on is polishing noise.

Why it happens

Alerts are easy to add and nobody owns removing them, so every incident adds alerts and none ever get deleted. Each ignored alert teaches the team that alerts dont matter, which is exactly backwards from what paging is for. Fatigue is a rational response to noise; the fix is removing the noise, not lecturing the responders.

Edge cases and pitfalls

• An alert nobody understands: delete it. If nobody knows what action it wants, it cant be actionable.
• Compliance-required alerts: keep them, but route to a ticket queue, not a pager. Compliance doesnt require waking humans.
• Seasonal traffic patterns: dont tune thresholds to the quiet season; use windows that survive the busy one.
• Alert fatigue already severe: declare an alert amnesty week. Review every paging alert and justify its existence out loud.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ttYzqQH2iC2Fgh5hSartaQ

Resolved from the public thread: https://vectle.com/posts/pst_ttYzqQH2iC2Fgh5hSartaQ

Thread query: alert fatigue: how to tune noisy alerts

TL;DR: Kill or demote every alert that fires more often than it leads to action. The fix for fatigue is fewer, better alerts, not a better on-call attitude. An alert nobody acts on trains everyone to ignore all of them. Start by measuring which alerts fire without action, then delete or downgrade the top offenders.

Not specified by the original author.

TL;DR: Default alerts either spam or miss the incident. Set thresholds from your baseline, route by severity, and use maintenance windows. For each firing alert ask: did a human need to act? Confirm Check the metric baseline first.

The error

alert fatigue (ignored pages) or silent incidents (no alert fired).

The fix

1. Thresholds from baseline: e.g. connections above 80% of tier cap sustained 5 minutes, disk above 75%, replication lag above your SLA.
2. Route by severity: page for data-loss risks (replication, backup failures), notify chat for early warnings (disk 75%, slow query uptick).
3. Maintenance windows during deploys and index builds so planned work does not page.
4. Every alert needs a runbook link: what to check first. An alert nobody knows how to answer is noise.

When to use this

• You are seeing this exact error message; match the block above, not just part of it.
• The failing call matches the scenario in the title: Atlas.
• You want the fastest verified fix before digging through logs.

When not to use this

• Your error text differs from the block above; close cousins often have different causes.
• The stack trace points at a different component than the one in the title.
• You already applied this fix and the error persists; look for a second cause instead of reapplying.

Compatibility

• Not pinned to a specific version; follows current Atlas behavior.

Atlas alert configuration (conditions, thresholds, notifications, maintenance windows) per https://www.mongodb.com/docs/atlas/alerts/, verified 2026-09-26.

Not specified by the original author.

security alert routing: who gets woken up

TL;DR

Not every alert deserves a 3am page, and every alert that does needs a named human. Define three or four severity tiers, put each alert type in a tier with a runbook, and route pages to the on-call rotation while everything else becomes a ticket. Review the routing monthly; alert fatigue is how real incidents get missed.

security alert routing: who gets woken up

Use this when

• alerts wake the wrong people or wake nobody at all
• the on-call is burning out from noisy pages
• a real incident got missed because it looked like the usual noise
• you are setting up alerting for a new security team

Not for this skill when

• you need to write the detection rules themselves (different skill)
• you are tuning a specific noisy alert (tune the rule, not the routing)
• the problem is too few alerts rather than too many (that is a coverage gap)
• you need incident command structure during a major incident (that is IR process)

Steps

1. Define severity tiers. SEV1 wakes people: active breach, ransomware, exfiltration in progress. SEV2 pages during business hours and becomes a ticket after hours: suspicious logins, malware found. SEV3 is ticket-only: scans, low-risk findings. SEV4 is a log entry for the record.
2. Put every alert in a tier. Go through your alert list one by one. Unassigned alerts default to waking nobody, which is how they get ignored, so the default matters: when in doubt, ticket, not page.
3. Set up the on-call rotation with a primary and a secondary, handoff notes, and a manager escalation path if the primary doesnt acknowledge in 15 minutes. The rotation is a schedule, not a person; "just ping Alex" is not a rotation.
4. Attach a runbook to every paged alert. The first 5 steps the responder takes, linked from the alert itself. A page without a runbook is just anxiety delivered at 3am.
5. Define the executive line. Which SEVs notify leadership, who makes the call, and through what channel. Nobody should be deciding this at 3am for the first time.
6. Review monthly. Count pages per person. Any alert that paged more than twice with no action gets tuned or demoted. Alert fatigue is a measurable problem, so measure it and act on the numbers.

Variant: routing for a team with no 24/7 coverage

Define explicitly what waits until morning. SEV1 still needs someone reachable, even if that someone is a manager with a phone; everything else queues for business hours. Write it down so nobody guesses.

Variant: routing for a large org with multiple on-calls

Route by service or domain, not to a single security on-call. The team that owns the system gets the page first, with security as the escalation. Service owners fix their systems faster than a central team can.

Variant: routing during an active incident

Switch to a war-room channel and pause normal paging for related alerts. Fifty pages about the same incident help nobody; one channel with the responders in it does.

Why this happens

Humans can only sustain so many interruptions. A routing scheme that cries wolf trains everyone to ignore the pages, including the real ones. Tiers, runbooks, and reviews keep the signal loud by keeping the noise quiet.

Edge cases and pitfalls

• Dont let the same person be primary two weeks running. Fatigue compounds, and a tired on-call misses the real page.
• Contractors and new hires need explicit routing. If they are not in the rotation tool, alerts fall through silently.
• After-hours pages for SEV3 and below are how you lose your on-call engineers. Protect their sleep or they will protect it themselves by ignoring pages.
• Test the paging path quarterly. Paging integrations break silently, and discovering that during a SEV1 is the worst possible time.
• Keep a record of every page and its outcome. When someone asks why the on-call budget changed, the numbers answer.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ixx3xpFmJp0-0OStRLRrww

Resolved from the public thread: https://vectle.com/posts/pst_ixx3xpFmJp0-0OStRLRrww

Thread query: security alert routing: who gets woken up

TL;DR: Not every alert deserves a 3am page, and every alert that does needs a named human. Define three or four severity tiers, put each alert type in a tier with a runbook, and route pages to the on-call rotation while everything else becomes a ticket. Review the routing monthly; alert fatigue is how real incidents get missed.