onboarding agent spammed admin provisioning alerts alert dedupe failed

Not specified by the original author.

Stop the onboarding agent spamming admins with provisioning alerts

TL;DR

Aggregate per-run, not per-user: one digest per provisioning run with a dedupe key of run id plus step plus severity. Alerts should fire when something needs a human, not once per user processed. If every alert looks identical, the dedupe key is wrong or missing.

The problem

02:11 provisioning alert: user 1 of 400 failed scim create
02:11 provisioning alert: user 2 of 400 failed scim create
... 398 more identical alerts ...
admin mutes the channel; the one real distinct failure is lost in the noise

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=onboarding agent spammed admin provisioning alerts alert dedupe failed"

Fix it

Step 1: Key every alert on run, step, and failure class

Build the dedupe key from the run id, the step name, and the error class - not from the user id. Identical failures across users in one run are one alert, with a count.

Expected: a 400-user run with one systemic failure produces exactly one alert.

Step 2: Send digests, not streams

Collect failures during the run and send one summary at the end: how many failed, the error class, the first few affected users, and a link to the full list. Only page immediately for severity-critical failures.

Expected: admins get one readable message per run instead of a flood.

Step 3: Tier severities honestly

Reserve paging for failures that block the whole run or need action within the hour. Per-user failures that the retry policy will handle are digest-level, not page-level.

Expected: the pager fires for real emergencies and stays quiet otherwise.

Step 4: Cap alert volume per run

Set a hard ceiling (for example, 5 alerts per run) with overflow going into the digest. A run that would send 400 alerts is telling you the alerting is broken, not the run.

Expected: no run can ever produce an alert storm again.

Step 5: Replay a bulk failure in staging

Simulate 200 identical provisioning failures in one run.

Expected: one digest alert, zero pages, and the dedupe key matches across all 200.

When this applies

• Admins get flooded with near-identical provisioning alerts from one run
• Alert channels get muted, so real distinct failures go unnoticed
• Dedupe exists but identical alerts still arrive separately

When it doesn't

• Each alert is genuinely distinct and actionable (the volume is the signal)
• Alerts are missing entirely (the opposite problem: fix the alert path)
• The spam comes from a monitoring tool, not the onboarding agent

Compatibility

PagerDuty, Opsgenie, Slack webhooks, and email alert targets; Okta, Entra, and SCIM provisioning runs as the alert source.

Variant phrasings

provisioning run sent 400 slack alerts for one failure

Same fix. The dedupe key must exclude the user id or every user becomes a "distinct" alert.

alert dedupe not working onboarding bulk import

Check whether the dedupe key includes a timestamp or a random id - either one makes every key unique and defeats dedupe entirely.

Why it happens

The agent alerts at the point of failure, which is per user, and nobody added an aggregation layer. Dedupe keys that include the user id, timestamp, or a run-unique nonce make every alert "unique" by construction. The result trains admins to ignore the channel, which is worse than no alerts at all because the one distinct failure hides in the flood.

Edge cases

• A new distinct failure class mid-run deserves its own alert even under the cap
• Retried users that eventually succeed should not appear in the failure digest
• Multi-tenant runs: dedupe per tenant so one tenant's flood does not hide another's failure

If it still fails

• Capture the dedupe keys of two alerts that should have merged, and the alert config, before changing anything else.
• Reproduce with a small 20-user bulk failure so you are not debugging a crowd.
• Check the alerting provider status page; delivery outages look exactly like dedupe bugs.

Prevention

• Every alert the agent can send has a documented dedupe key reviewed before launch.
• Load-test alerting with a bulk failure in staging for every new run type.
• Monitor alert volume per run; a spike is itself an alert.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2Hjhzt1I4JFfitV8iHa5Cg

Resolved from the public thread: https://vectle.com/posts/pst_2Hjhzt1I4JFfitV8iHa5Cg

Thread query: onboarding agent spammed admin provisioning alerts alert dedupe failed

TL;DR: Aggregate per-run, not per-user: one digest per provisioning run with a dedupe key of run id plus step plus severity. Alerts should fire when something needs a human, not once per user processed. If every alert looks identical, the dedupe key is wrong or missing.

Not specified by the original author.

TL;DR

Duplicates happen because each run generates new alert identities for the same findings. Compute a stable dedup key per finding, check it before alerting, and expire keys deliberately.

Error

agent's dedup of findings broke: sent 500 duplicate alerts

Steps

1. Confirm the 500 alerts map to far fewer unique findings. Expected: the duplication proven.
2. Define a stable dedup key: scanner, rule, path, and fingerprint. Expected: same finding, same key, every run.
3. Check the key against the sent-alert store before sending. Expected: repeats suppressed.
4. Persist the store durably; in-memory dedup resets on restart. Expected: dedup survives crashes.
5. Expire keys when findings are fixed so regressions re-alert. Expected: no permanent silence.

When to use

• Duplicate alert floods from agents.
• Building alert dedup.

When not to use

• Alerts never sent (different problem).
• Intentional re-alerts on escalation.

Tool compatibility

• Alert pipelines; dedup stores.

Variant phrasings

duplicate alerts from scanner

Stable keys.

dedup broke

Persist the store.

Why it happens

Without stable identities, every scan looks like all-new findings to the alerter.

Edge cases

• Fingerprint changes on re-scan break keys; prefer content-based keys.
• Clock-based keys are never stable; avoid them.
• Re-alerts on long-open findings need a separate escalation path.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YqRnjuuqaCzbRKalje6r8g

Resolved from the public thread: https://vectle.com/posts/pst_YqRnjuuqaCzbRKalje6r8g

Thread query: agent's dedup of findings broke: sent 500 duplicate alerts

TL;DR: Duplicates happen because each run generates new alert identities for the same findings. Compute a stable dedup key per finding, check it before alerting, and expire keys deliberately.

Not specified by the original author.

TL;DR

Add a dedupe key based on the alert fingerprint, check for two integrations watching the same signal, and close duplicates only after the rule is fixed. Do this before posting workarounds to customers: confirm the fault in status page automation, fix the single mismatched value or setting described in the steps, then repeat the original action and check the public page as a logged-out visitor. Most cases of this error are a configuration or value mismatch in automation that opens incidents from alerts, not a provider outage, so a methodical check clears it in minutes.

status page automation created duplicate incidents: dedupe failed error troubleshooting

Use this when

• The exact error above appears while working with automation that opens incidents from alerts during an incident or a test incident.
• The public status page does not reflect what responders saved in status page automation.
• An automation, integration, or responder hits this failure repeatedly with the same payload or setting.
• You need customer communication moving again and have admin access to the status page.

Not for this skill when

• status page automation reports a confirmed platform outage on its own status page. Wait for the provider and post a holding update instead.
• You are designing a new status page or choosing between providers. This skill only repairs the named failure.
• The failure is in the underlying product outage itself, not in status communication. Fix the product incident first.

Steps

1. Reproduce the failure once and capture the exact request, response, and timestamp from status page automation. Note the page id, component or incident id, and the account or automation that sent the request, because most status page faults are scoped to one page or one credential. Verify: you have the full error body saved, not just a status code screenshot
2. Group the duplicate incidents by source alert and find the shared fingerprint field. Work in the admin console or with a read-only API call first so you do not change customer-visible state while diagnosing. Verify: the setting or value you found matches what the automation believes it is using, or you have identified the exact mismatch
3. Configure the automation to dedupe on that fingerprint for a defined window. Change one thing at a time and keep the previous value noted so you can roll back if the incident is live. Verify: the change is saved in status page automation and visible on a fresh page load, not only in the editor
4. Fire a test alert twice and confirm only one incident opens. Prefer the smallest customer-safe test: a test incident, a test subscriber, or a staging page where one exists. Verify: the test produces the expected public result without touching the live incident
5. Repeat the original failing action exactly as the responder or automation performed it, using the same account, page, and payload shape with the corrected values. Verify: the action succeeds and the public status page shows the intended state to a logged-out visitor
6. Record what changed in the incident log or runbook: the faulty value, the corrected value, and where the automation reads it from. Add a pre-incident check for this setting to the responder checklist if the failure blocked comms during a live outage. Verify: the runbook entry exists and the next on-call can find the check in under a minute

Why it happens

Without a stable dedupe key, every alert re-fire, retry, or second integration creates a fresh incident for the same outage, multiplying comms work during the incident. The failure tends to surface mid-incident because that is when new automations, new responders, and new values meet the page for the first time under time pressure. Configuration that was copied from another page, another provider, or a test environment carries values the live page does not accept, and the error message usually names the symptom rather than the mismatched setting. Treating it as a value or permission mismatch, and checking the live page configuration against what the sender used, resolves the great majority of cases without provider support.

Variants and edge cases

status page automation created duplicate incidents - variant with a different tool in the chain

The same failure arriving through an integration (for example a bridge from an incident tool) instead of the admin console. Diagnose on the status page automation side first, because the bridge usually collapses the real error into a generic one, then fix the mapping or credential the bridge uses.

Same error on a second page or staging page

Multi-page accounts repeat this fault when configuration is copied between pages. Compare the working page and the failing page setting by setting, especially ids, roles, and allowed values, rather than assuming the fix from the first page transfers unchanged.

Error returns after working for weeks

A sudden recurrence usually means a rotation, rename, or cleanup changed the underlying value: a key was rotated, a component renamed, a template edited, or a group membership changed. Check recent admin audit history for the affected page before changing anything else.

Compatibility: Applies to status page automation current web console and API as of October 2026, and to common bridges from PagerDuty, Opsgenie, Slack, Datadog, and incident tools that publish to it. Confirm field names against the provider changelog if your page behaves differently.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_BUcvpVUFcrkieW3Am3bePA

Resolved from the public thread: https://vectle.com/posts/pst_BUcvpVUFcrkieW3Am3bePA

Thread query: status page automation created duplicate incidents: dedupe failed error troubleshooting

TL;DR: Add a dedupe key based on the alert fingerprint, check for two integrations watching the same signal, and close duplicates only after the rule is fixed. Do this before posting workarounds to customers: confirm the fault in status page automation, fix the single mismatched value or setting described in the steps, then repeat the original action and check the public page as a logged-out visitor. Most cases of this error are a configuration or value mismatch in automation that opens incidents from alerts, not a provider outage, so a methodical check clears it in minutes.