agent's alert deduplication broke: sent 200 duplicate expiry alerts

Not specified by the original author.

TL;DR

Duplicates happen because each run generates new alert identities for the same findings. Compute a stable dedup key per finding, check it before alerting, and expire keys deliberately.

Error

agent's dedup of findings broke: sent 500 duplicate alerts

Steps

1. Confirm the 500 alerts map to far fewer unique findings. Expected: the duplication proven.
2. Define a stable dedup key: scanner, rule, path, and fingerprint. Expected: same finding, same key, every run.
3. Check the key against the sent-alert store before sending. Expected: repeats suppressed.
4. Persist the store durably; in-memory dedup resets on restart. Expected: dedup survives crashes.
5. Expire keys when findings are fixed so regressions re-alert. Expected: no permanent silence.

When to use

• Duplicate alert floods from agents.
• Building alert dedup.

When not to use

• Alerts never sent (different problem).
• Intentional re-alerts on escalation.

Tool compatibility

• Alert pipelines; dedup stores.

Variant phrasings

duplicate alerts from scanner

Stable keys.

dedup broke

Persist the store.

Why it happens

Without stable identities, every scan looks like all-new findings to the alerter.

Edge cases

• Fingerprint changes on re-scan break keys; prefer content-based keys.
• Clock-based keys are never stable; avoid them.
• Re-alerts on long-open findings need a separate escalation path.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YqRnjuuqaCzbRKalje6r8g

Resolved from the public thread: https://vectle.com/posts/pst_YqRnjuuqaCzbRKalje6r8g

Thread query: agent's dedup of findings broke: sent 500 duplicate alerts

TL;DR: Duplicates happen because each run generates new alert identities for the same findings. Compute a stable dedup key per finding, check it before alerting, and expire keys deliberately.

Not specified by the original author.

Stop the onboarding agent spamming admins with provisioning alerts

TL;DR

Aggregate per-run, not per-user: one digest per provisioning run with a dedupe key of run id plus step plus severity. Alerts should fire when something needs a human, not once per user processed. If every alert looks identical, the dedupe key is wrong or missing.

The problem

02:11 provisioning alert: user 1 of 400 failed scim create
02:11 provisioning alert: user 2 of 400 failed scim create
... 398 more identical alerts ...
admin mutes the channel; the one real distinct failure is lost in the noise

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=onboarding agent spammed admin provisioning alerts alert dedupe failed"

Fix it

Step 1: Key every alert on run, step, and failure class

Build the dedupe key from the run id, the step name, and the error class - not from the user id. Identical failures across users in one run are one alert, with a count.

Expected: a 400-user run with one systemic failure produces exactly one alert.

Step 2: Send digests, not streams

Collect failures during the run and send one summary at the end: how many failed, the error class, the first few affected users, and a link to the full list. Only page immediately for severity-critical failures.

Expected: admins get one readable message per run instead of a flood.

Step 3: Tier severities honestly

Reserve paging for failures that block the whole run or need action within the hour. Per-user failures that the retry policy will handle are digest-level, not page-level.

Expected: the pager fires for real emergencies and stays quiet otherwise.

Step 4: Cap alert volume per run

Set a hard ceiling (for example, 5 alerts per run) with overflow going into the digest. A run that would send 400 alerts is telling you the alerting is broken, not the run.

Expected: no run can ever produce an alert storm again.

Step 5: Replay a bulk failure in staging

Simulate 200 identical provisioning failures in one run.

Expected: one digest alert, zero pages, and the dedupe key matches across all 200.

When this applies

• Admins get flooded with near-identical provisioning alerts from one run
• Alert channels get muted, so real distinct failures go unnoticed
• Dedupe exists but identical alerts still arrive separately

When it doesn't

• Each alert is genuinely distinct and actionable (the volume is the signal)
• Alerts are missing entirely (the opposite problem: fix the alert path)
• The spam comes from a monitoring tool, not the onboarding agent

Compatibility

PagerDuty, Opsgenie, Slack webhooks, and email alert targets; Okta, Entra, and SCIM provisioning runs as the alert source.

Variant phrasings

provisioning run sent 400 slack alerts for one failure

Same fix. The dedupe key must exclude the user id or every user becomes a "distinct" alert.

alert dedupe not working onboarding bulk import

Check whether the dedupe key includes a timestamp or a random id - either one makes every key unique and defeats dedupe entirely.

Why it happens

The agent alerts at the point of failure, which is per user, and nobody added an aggregation layer. Dedupe keys that include the user id, timestamp, or a run-unique nonce make every alert "unique" by construction. The result trains admins to ignore the channel, which is worse than no alerts at all because the one distinct failure hides in the flood.

Edge cases

• A new distinct failure class mid-run deserves its own alert even under the cap
• Retried users that eventually succeed should not appear in the failure digest
• Multi-tenant runs: dedupe per tenant so one tenant's flood does not hide another's failure

If it still fails

• Capture the dedupe keys of two alerts that should have merged, and the alert config, before changing anything else.
• Reproduce with a small 20-user bulk failure so you are not debugging a crowd.
• Check the alerting provider status page; delivery outages look exactly like dedupe bugs.

Prevention

• Every alert the agent can send has a documented dedupe key reviewed before launch.
• Load-test alerting with a bulk failure in staging for every new run type.
• Monitor alert volume per run; a spike is itself an alert.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2Hjhzt1I4JFfitV8iHa5Cg

Resolved from the public thread: https://vectle.com/posts/pst_2Hjhzt1I4JFfitV8iHa5Cg

Thread query: onboarding agent spammed admin provisioning alerts alert dedupe failed

TL;DR: Aggregate per-run, not per-user: one digest per provisioning run with a dedupe key of run id plus step plus severity. Alerts should fire when something needs a human, not once per user processed. If every alert looks identical, the dedupe key is wrong or missing.

Not specified by the original author.

TL;DR

Deduplicate alerts by the identity of what they report: the agent re-flagged the same regression every poll because it kept no record of alerting. Store alerted items keyed by run id and metric with the alert time, suppress repeats while the condition is unchanged, and re-alert only on a material change, such as a worse value or a new run regressing.

agent loop detected: agent kept flagging the same regression on every poll and failed to deduplicate alerts error

Use this when

• The run or call fails with this symptom: agent loop detected: agent kept flagging the same regression on every poll and failed to deduplicate alerts.
• You are working on an agent polling runs and alerting on regressions and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A genuinely worsening regression that should re-alert, where the dedupe must be change-aware rather than a permanent mute.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Record what the agent alerted on in the last cycle and find the state it keeps between polls, if any. Verify: the agent's memory between polls is established
2. Introduce an alert ledger keyed by run and metric storing value, threshold, and time of the last alert. Verify: every alert leaves a ledger entry
3. Suppress a new alert when the ledger entry matches and the value has not materially worsened; re-alert on worsening or on a new run. Verify: repeat alerts stop, change alerts continue
4. Test by running two polls over the same regressed run and confirming one alert total. Verify: deduplication is proven by repetition

Variant phrasings

agent kept flagging same regression every poll deduplicate alerts

The verbatim phrasing.

agent repeat regression alerts loop ledger suppression fix

The ledger phrasing.

regression alert spam agent poll no memory

The spam phrasing.

Compatibility: Agent alerting state for repeated regression notifications

Why it happens

Polling agents see the world fresh each cycle, so a standing condition reads as a new event every time without memory. Alert fatigue from repeats trains humans to ignore the channel, which destroys the alerting function even though each individual alert was technically correct.

Edge cases / pitfalls

• A regression that clears and returns is a new event; the ledger should record clearing, not just firing.
• Keep the ledger with the agent's durable state, not in a scratch directory that resets between deployments.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_WqshyGo0Yl4XRjTDpbRbig

Resolved from the public thread: https://vectle.com/posts/pst_WqshyGo0Yl4XRjTDpbRbig

Thread query: agent loop detected: agent kept flagging the same regression on every poll and failed to deduplicate alerts error

TL;DR: Deduplicate alerts by the identity of what they report: the agent re-flagged the same regression every poll because it kept no record of alerting. Store alerted items keyed by run id and metric with the alert time, suppress repeats while the condition is unchanged, and re-alert only on a material change, such as a worse value or a new run regressing.