fraud agent failed under alert fatigue: 200 low-confidence flags
Cures alert fatigue from floods of low-confidence flags with severity tiers, hourly digests, and auto-close for the lowest tier. Use when analysts ignore alerts because volume makes each one cheap. Not for genuinely high attack volume or low-precision scorers.
TL;DR
Two hundred low-confidence flags is a tuning failure, not a people failure. Stop sending raw flags to humans: bucket them into severity tiers, auto-close the lowest tier with a logged reason, and only page on tier one. If the tier-one count stays sane while coverage holds, the fatigue goes away without losing real signal.
fraud agent failed under alert fatigue: 200 low-confidence flagsUse this when
- Analysts ignore alerts because volume makes each one look cheap.
- Most flags are low confidence and rarely convert.
- The agent pages or pings for every flag instead of batching.
Not for this skill when
- Alert volume is high because attacks are actually high. That is a staffing question, or a block rule.
- Flags are wrong, not just low confidence. Fix precision first.
Steps
- Measure conversion per confidence band for the last 30 days. Verify: you can quote the close rate of flags under 0.5 confidence.
- Define tiers: tier one pages immediately, tier two batches hourly into a digest, tier three auto-closes with a logged reason code. Verify: a tier-three flag produces a log line, not a notification.
- Make the agent summarize tier two in one digest per hour with counts and top entities, never one message per flag. Verify: 200 flags produce one digest, not 200 messages.
- Recheck weekly: if tier-one precision drops, retune the band edges. Verify: a dashboard shows alerts per tier and conversion per tier side by side.
Variant phrasings
too many low confidence fraud alerts
Volume phrasing.
analyst alert fatigue fraud agent
People-impact phrasing.
fraud agent spamming low score flags
Agent-behavior phrasing.
Compatibility: Fits any alerting pipeline with numeric confidence. Works whether notifications go to chat, a pager, or email digests.
Why it happens
Thresholds get set once and never revisited, models get noisier under drift, and every flag costs the agent nothing to send. Humans pay the cost, so they stop reading, and the one real flag hides in the noise.
Edge cases / pitfalls
- Auto-closing needs a sampling audit, or a whole fraud class can go dark silently.
- Digests need dedupe by entity or one ring shows up fifty times in one email.
- Do not let tier three include account-takeover patterns. Some classes never auto-close.
- Track the false-negative rate on a holdout sample, not just precision, or coverage quietly dies.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_jIteQoZ0hjgmg-pO-eiTRQ