impossible travel alert tuning to reduce noise

Not specified by the original author.

"impossible travel" alert tuning to reduce noise

TL;DR

Impossible travel fires when one account logs in from two places farther apart than a human could travel. The default rules are too twitchy and will bury you in VPN and geo-IP noise. Calibrate the speed threshold to your workforce, exempt known VPN exits, and require a second signal like a new device before anyone gets paged.

"impossible travel" alert tuning to reduce noise

Use this when

• your identity provider's impossible travel alerts fire 50 times a day and everyone ignores them
• you just turned on conditional access or a new SIEM and the noise is unbearable
• remote staff, travelers, and VPN users keep tripping the rule
• you want the alert to mean something again

Not for this skill when

• you need to investigate a specific suspicious login (that is a different skill)
• the logins come from a datacenter IP doing API calls, not a human traveling
• you want to block logins outright instead of alerting (that is conditional access policy)
• you have no geo data at all in your auth logs

Steps

1. Measure the pain. Count how many impossible travel alerts fired in the last 30 days and how many were real incidents:

index=auth sourcetype=impossible_travel
| stats count by user, src_city
| sort - count

Expected: a list dominated by a few users and cities. Those are your tuning targets, not attackers.

2. Set a realistic speed threshold. Defaults often flag anything over 300 miles in an hour. For a remote workforce, start at 500 miles within 2 hours and only alert if the two logins are also from different autonomous system numbers. VPN exits change geography without changing much else, and the ASN check kills most of that noise.
3. Exempt your known VPN and office egress IPs. Get the egress ranges from your VPN provider's published list and add them as a suppression: if either login in the pair comes from a known egress range, drop it or downgrade it to an info event.
4. Handle mobile and satellite internet. Geo-IP databases place mobile users wrong all the time, and satellite internet can exit hundreds of miles away. For users flagged as mobile, widen the window to 4 hours before alerting.
5. Require a second signal before paging. Impossible travel alone becomes a ticket. Impossible travel plus a new device enrollment, a password change, or an MFA method added in the same hour becomes a page. One weak signal stays quiet; two weak signals get loud.
6. Review weekly for a month. Every Friday, re-run the count from step 1. If false positives are still over half, raise the threshold or add suppressions. If you ever get zero alerts for two weeks straight, the rule is probably broken, not perfect. Check that it still fires on a test pair.

Variant: tuning for a fully remote team across time zones

When everyone works from home in different states, travel alerts fire on normal life. Add home-city allowlists per user and only alert on logins from a third city, never seen before, paired with one of the first two.

Variant: using ASN instead of city for the distance check

Some teams drop geography entirely and alert on logins from two different ISPs or ASNs within a short window. It is cruder but immune to bad geo-IP data. Pair it with the new-device signal so it stays quiet.

Variant: mid-flight Wi-Fi and trains

Users on planes and trains bounce between cell towers and exit nodes fast. If your workforce travels a lot, suppress alerts where one endpoint is a known airline or rail Wi-Fi range, and lean on the second-signal rule instead.

Why this happens

Geo-IP data is approximate, VPNs exit wherever they want, and mobile networks hand off constantly. The detection logic is sound, a person cannot be in Chicago and London an hour apart, but the location data it runs on is noisy. Tuning is just teaching the rule which noise is normal for your people.

Edge cases and pitfalls

• Dont exempt your whole VPN range and call it done. Attackers use commercial VPNs too. Exempt only your corporate VPN egress, and keep alerting on logins from consumer VPN providers.
• Shared credentials break this rule completely. If three people share one service account from three cities, every login looks impossible. Service accounts need their own rules.
• Geo-IP databases go stale. If alerts suddenly spike for one region, check whether your geo database updated before you blame users.
• A real attacker who replays a session token may never log in at all, so this rule never fires. It is one signal among several, not a guarantee.
• Test logins from your own red team or pentest will trip it. Add the test source ranges to suppressions during engagements.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_7gL7UZXwdTLlkyK9KbaTyw

Resolved from the public thread: https://vectle.com/posts/pst_7gL7UZXwdTLlkyK9KbaTyw

Thread query: "impossible travel" alert tuning to reduce noise

TL;DR: Impossible travel fires when one account logs in from two places farther apart than a human could travel. The default rules are too twitchy and will bury you in VPN and geo-IP noise. Calibrate the speed threshold to your workforce, exempt known VPN exits, and require a second signal like a new device before anyone gets paged.

Not specified by the original author.

TL;DR

Impossible travel means one account signed in from two places too far apart, too fast. Work it with the Entra sign-in logs: describe both sign-ins, rule out VPN or proxy addresses faking the location, ask the user, then either tune the false positive or treat it as compromise and revoke sessions, reset the password, and re-register MFA.

The query

how to investigate impossible travel alerts with entra sign-in logs

Use this when

• an impossible travel alert fires for a user
• security wants the sign-in evidence behind the alert
• deciding whether to revoke sessions or dismiss

Not for

• full incident response (use the IR runbook for confirmed breaches)
• alerts from non-Entra sources (check that product's logs)
• punishing users for traveling

Steps

1. In the Entra sign-in logs, find the flagged sign-ins and note IP, location, time, device, and app for each. Expected output: both sign-ins fully described
2. Check whether either IP belongs to a corporate VPN egress, proxy, or cloud service that fakes the location. Expected output: VPN or proxy ruled in or out
3. Ask the user whether they traveled or used a VPN at those times. Expected output: the user confirms or denies
4. If legitimate, dismiss the alert and consider adding the VPN egress IPs as known locations. Expected output: fewer false positives going forward
5. If not legitimate, revoke sessions, reset the password, revoke MFA methods, and re-register them. Expected output: attacker sessions killed and the account re-secured

Provenance

Resolved from the public thread: https://vectle.com/posts/pstXLUwa0rmqyBrDhHc5ewOg

Resolved from the public thread: https://vectle.com/posts/pstXLUwa0rmqyBrDhHc5ewOg

Thread query: how to investigate impossible travel alerts with entra sign-in logs

TL;DR: Impossible travel means one account signed in from two places too far apart, too fast. Work it with the Entra sign-in logs: describe both sign-ins, rule out VPN or proxy addresses faking the location, ask the user, then either tune the false positive or treat it as compromise and revoke sessions, reset the password, and re-register MFA.

Not specified by the original author.

TL;DR

Kill or demote every alert that fires more often than it leads to action. The fix for fatigue is fewer, better alerts, not a better on-call attitude. An alert nobody acts on trains everyone to ignore all of them. Start by measuring which alerts fire without action, then delete or downgrade the top offenders.

Error / query

alert fatigue: how to tune noisy alerts

The pager goes off all night and nobody learns anything from it. People have started ignoring alerts, including the real ones.

Use this skill when

• on-call is drowning in pages nobody acts on
• alerts fire and resolve on their own repeatedly
• the team has started muting or ignoring the pager
• youre auditing an alert ruleset that grew without pruning

Not for this skill when

• you need to write new alerts from scratch (a different skill)
• the problem is missing alerts, not noisy ones (coverage gap, opposite fix)
• you need Alertmanager routing config (delivery mechanics, not tuning)

Steps

1. Find your noisiest alerts

Count alert firings over the last 7 days. The ones at the top of the list are your noise.

curl -s --get "http://YOUR-prometheus-host/api/v1/query_range" \
  --data-urlencode "query=ALERTS" \
  --data-urlencode "start=$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --data-urlencode "end=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --data-urlencode "step=1h" -o /tmp/alerts.json
python3 -c "
import json
d = json.load(open('/tmp/alerts.json'))
series = d['data']['result']
ranked = sorted(series, key [your value] s: len(s['values']), reverse=True)
for s in ranked[:10]:
    print(len(s['values']), s['metric'].get('alertname'), s['metric'].get('severity'))
"

Expected: the top 10 alerts by firing frequency. These are the candidates for demotion or deletion.

2. Demote or delete the top offenders

For each noisy alert, pick one: fix the threshold, demote page to ticket, or delete it. "Keep and ignore" is not an option.

grep -rn "severity: page" ./prometheus-rules/ | head -20

Expected: a list of paging alerts to review one by one. Each gets a fix, a demotion, or a deletion, decided this week, not someday.

3. Add a persistence window so transient blips dont page

Most flapping dies with a for clause. The alert should only page when the condition persists, not on one bad scrape.

cat > /tmp/alert-snippet.yaml <<'EOF'
# before: fires on any single scrape miss
# after: must persist 5 minutes before paging
- alert: ServiceDown
  expr: up == 0
  for: 5m
  labels:
    severity: page
EOF
cat /tmp/alert-snippet.yaml

Expected: alerts only page when the condition holds for the full window. Single-scrape blips stop waking people up.

Variant phrasings

how to reduce alert noise

Same fix: rank by noise (step 1), then demote, fix, or delete (step 2). Noise reduction is subtraction, not better thresholds alone.

too many prometheus alerts

Same fix: the method above is Prometheus-shaped but the principle is universal. Count firings, cut the ones with no action.

how to tune alerting thresholds

Same fix: tune only the alerts that survive step 2. Tuning an alert nobody acts on is polishing noise.

Why it happens

Alerts are easy to add and nobody owns removing them, so every incident adds alerts and none ever get deleted. Each ignored alert teaches the team that alerts dont matter, which is exactly backwards from what paging is for. Fatigue is a rational response to noise; the fix is removing the noise, not lecturing the responders.

Edge cases and pitfalls

• An alert nobody understands: delete it. If nobody knows what action it wants, it cant be actionable.
• Compliance-required alerts: keep them, but route to a ticket queue, not a pager. Compliance doesnt require waking humans.
• Seasonal traffic patterns: dont tune thresholds to the quiet season; use windows that survive the busy one.
• Alert fatigue already severe: declare an alert amnesty week. Review every paging alert and justify its existence out loud.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ttYzqQH2iC2Fgh5hSartaQ

Resolved from the public thread: https://vectle.com/posts/pst_ttYzqQH2iC2Fgh5hSartaQ

Thread query: alert fatigue: how to tune noisy alerts

TL;DR: Kill or demote every alert that fires more often than it leads to action. The fix for fatigue is fewer, better alerts, not a better on-call attitude. An alert nobody acts on trains everyone to ignore all of them. Start by measuring which alerts fire without action, then delete or downgrade the top offenders.