security alert routing: who gets woken up

security alert routing: who gets woken up

TL;DR

Not every alert deserves a 3am page, and every alert that does needs a named human. Define three or four severity tiers, put each alert type in a tier with a runbook, and route pages to the on-call rotation while everything else becomes a ticket. Review the routing monthly; alert fatigue is how real incidents get missed.

security alert routing: who gets woken up

Use this when

• alerts wake the wrong people or wake nobody at all
• the on-call is burning out from noisy pages
• a real incident got missed because it looked like the usual noise
• you are setting up alerting for a new security team

Not for this skill when

• you need to write the detection rules themselves (different skill)
• you are tuning a specific noisy alert (tune the rule, not the routing)
• the problem is too few alerts rather than too many (that is a coverage gap)
• you need incident command structure during a major incident (that is IR process)

Steps

1. Define severity tiers. SEV1 wakes people: active breach, ransomware, exfiltration in progress. SEV2 pages during business hours and becomes a ticket after hours: suspicious logins, malware found. SEV3 is ticket-only: scans, low-risk findings. SEV4 is a log entry for the record.

1. Put every alert in a tier. Go through your alert list one by one. Unassigned alerts default to waking nobody, which is how they get ignored, so the default matters: when in doubt, ticket, not page.

1. Set up the on-call rotation with a primary and a secondary, handoff notes, and a manager escalation path if the primary doesnt acknowledge in 15 minutes. The rotation is a schedule, not a person; "just ping Alex" is not a rotation.

1. Attach a runbook to every paged alert. The first 5 steps the responder takes, linked from the alert itself. A page without a runbook is just anxiety delivered at 3am.

1. Define the executive line. Which SEVs notify leadership, who makes the call, and through what channel. Nobody should be deciding this at 3am for the first time.

1. Review monthly. Count pages per person. Any alert that paged more than twice with no action gets tuned or demoted. Alert fatigue is a measurable problem, so measure it and act on the numbers.

Variant: routing for a team with no 24/7 coverage

Define explicitly what waits until morning. SEV1 still needs someone reachable, even if that someone is a manager with a phone; everything else queues for business hours. Write it down so nobody guesses.

Variant: routing for a large org with multiple on-calls

Route by service or domain, not to a single security on-call. The team that owns the system gets the page first, with security as the escalation. Service owners fix their systems faster than a central team can.

Variant: routing during an active incident

Switch to a war-room channel and pause normal paging for related alerts. Fifty pages about the same incident help nobody; one channel with the responders in it does.

Why this happens

Humans can only sustain so many interruptions. A routing scheme that cries wolf trains everyone to ignore the pages, including the real ones. Tiers, runbooks, and reviews keep the signal loud by keeping the noise quiet.

Edge cases and pitfalls

• Dont let the same person be primary two weeks running. Fatigue compounds, and a tired on-call misses the real page.
• Contractors and new hires need explicit routing. If they are not in the rotation tool, alerts fall through silently.
• After-hours pages for SEV3 and below are how you lose your on-call engineers. Protect their sleep or they will protect it themselves by ignoring pages.
• Test the paging path quarterly. Paging integrations break silently, and discovering that during a SEV1 is the worst possible time.
• Keep a record of every page and its outcome. When someone asks why the on-call budget changed, the numbers answer.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ixx3xpFmJp0-0OStRLRrww