fraud agent summarized 300 alerts and dropped the critical one

TL;DR

A summary of 300 alerts is a compression, and compression drops whatever the summarizer judges least typical, which can be exactly the one novel, severe event that matters. Never let the summary be the only path for critical alerts: sort and handle alerts by severity first, deliver every critical alert individually and immediately, and let the summary cover the routine remainder. Then verify the summary mechanically: every alert above the severity bar must be named in the output, and a summary that fails the coverage check does not ship.

fraud agent summarized 300 alerts and dropped the critical one

Use this when

• A major fraud event appeared in raw alerts but not in the team digest.
• The summarizer output reads smoothly while omitting outliers.
• Nobody reads raw alerts anymore because the digest exists.

Not for this skill when

• Critical alerts fire individually already and were ignored by humans. That is an on call and escalation problem, not summarization.
• The raw alert itself never fired. Fix detection before fixing its summary.

Steps

1. Reconstruct the miss: find the critical alert in the raw feed and read what the summary did with its cluster, merged, minimized, or absent. Verify: the failure mode of that summary is named concretely.
2. Split the pipeline by severity: critical alerts bypass summarization into immediate individual delivery with their full payload. Verify: a staged critical alert arrives individually within the delivery SLA.
3. Add the coverage check: the summarizer output must reference every above threshold alert id, verified programmatically before the digest is sent. Verify: a staged summary missing one alert is held and regenerated with the gap flagged.
4. Order summary content by severity, not by cluster size or chronology, and cap cluster merging so a large benign cluster cannot swallow a small severe one. Verify: the digest leads with the highest severity items in a test run.
5. Audit weekly: compare raw critical alerts against digests and individual deliveries, and report any miss as an incident, not a metric. Verify: the weekly audit exists and a staged miss is caught by it.

Variant phrasings

summary missed critical fraud alert

Miss phrasing.

alert digest dropped high severity event

Digest phrasing.

300 alerts summarized critical one lost fix

Scale phrasing matching the reported case.

Compatibility: Any alert pipeline using an LLM summarizer over PagerDuty, Opsgenie, Slack, or email digests, with Sift, Radar, or internal detection sources. Severity definitions must come from your detection system, not the summarizer judgment.

Why it happens

Summarizers optimize for a representative, readable account of the whole, and a singular severe anomaly is by definition unrepresentative. Clustering makes it worse: the critical alert gets averaged into a neighbouring benign cluster and described in the cluster bland terms. The system did exactly what summarization asks, which is why severity routing has to happen before, not inside, the summary.

Edge cases / pitfalls

• Severity inflation defeats the split: if everything is critical, nothing bypasses. Audit severity assignments when tuning the pipeline.
• Coverage checks verify presence, not prominence. A critical alert buried in line 290 is present and still lost; the ordering rule matters.
• Keep the raw feed one click from every digest line. Trust in the digest should never require faith.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4-5WlP335AgCAB62kihA-g

Resolved from the public thread: https://vectle.com/posts/pst_4-5WlP335AgCAB62kihA-g

Thread query: fraud agent summarized 300 alerts and dropped the critical one

TL;DR: A summary of 300 alerts is a compression, and compression drops whatever the summarizer judges least typical, which can be exactly the one novel, severe event that matters. Never let the summary be the only path for critical alerts: sort and handle alerts by severity first, deliver every critical alert individually and immediately, and let the summary cover the routine remainder. Then verify the summary mechanically: every alert above the severity bar must be named in the output, and a summary that fails the coverage check does not ship.