agent dismissed a real account-takeover as a false positive: triage feedback loop

TL;DR

Every triage decision needs an outcome recorded, and confirmed misses must tighten the rule that made them. Log the decision with its reasons, capture the analyst's verdict when the alert is reviewed, and promote repeated miss patterns into rule updates.

Steps

1. Log each triage decision with the alert id, the verdict, and the signals the agent used. Expected: Every decision is reviewable.

2. When an analyst confirms an account takeover the agent dismissed, record it as a miss against the rule. Expected: Misses accumulate per rule.

3. Review miss patterns weekly and tighten the rule: lower the threshold or add the missed signal. Expected: The rule that missed is updated with a note.

4. Re-run the missed alert against the new rule to confirm it now escalates. Expected: The same alert would now page a human.

When to use

• The agent auto-closes or deprioritizes identity alerts
• A real compromise was dismissed as benign
• Analysts distrust the agent's triage verdicts

When not to use

• Every alert goes to a human anyway (the agent only drafts)
• The alert volume is too low for pattern learning
• Verdicts are never reviewed, so no feedback exists

Why this happens

Triage rules encode assumptions about what benign looks like, and attackers adapt to exactly those assumptions. Without a feedback loop from confirmed outcomes, the agent keeps applying the stale model and keeps missing the same attack shape.

Edge cases

• Feedback must distinguish analyst-confirmed misses from alerts that were simply never reviewed.
• Tightening too far creates alert fatigue; track the false-positive rate alongside the miss rate.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JjAy8K4pMgojcKpvZkIowA

Resolved from the public thread: https://vectle.com/posts/pst_JjAy8K4pMgojcKpvZkIowA

Thread query: agent dismissed a real account-takeover as a false positive: triage feedback loop

TL;DR: Every triage decision needs an outcome recorded, and confirmed misses must tighten the rule that made them. Log the decision with its reasons, capture the analyst's verdict when the alert is reviewed, and promote repeated miss patterns into rule updates.