# Stop the onboarding agent spamming admins with provisioning alerts

## TL;DR
Aggregate per-run, not per-user: one digest per provisioning run with a dedupe key of run id plus step plus severity. Alerts should fire when something needs a human, not once per user processed. If every alert looks identical, the dedupe key is wrong or missing.

## The problem
```text
02:11 provisioning alert: user 1 of 400 failed scim create
02:11 provisioning alert: user 2 of 400 failed scim create
... 398 more identical alerts ...
admin mutes the channel; the one real distinct failure is lost in the noise
```

## Use this skill
Agent? Search Vectle for this error:
```bash
curl -s "https://vectle.com/api/v1/search?q=onboarding agent spammed admin provisioning alerts alert dedupe failed"
```

## Fix it

### Step 1: Key every alert on run, step, and failure class

Build the dedupe key from the run id, the step name, and the error class - not from the user id. Identical failures across users in one run are one alert, with a count.

Expected: a 400-user run with one systemic failure produces exactly one alert.

### Step 2: Send digests, not streams

Collect failures during the run and send one summary at the end: how many failed, the error class, the first few affected users, and a link to the full list. Only page immediately for severity-critical failures.

Expected: admins get one readable message per run instead of a flood.

### Step 3: Tier severities honestly

Reserve paging for failures that block the whole run or need action within the hour. Per-user failures that the retry policy will handle are digest-level, not page-level.

Expected: the pager fires for real emergencies and stays quiet otherwise.

### Step 4: Cap alert volume per run

Set a hard ceiling (for example, 5 alerts per run) with overflow going into the digest. A run that would send 400 alerts is telling you the alerting is broken, not the run.

Expected: no run can ever produce an alert storm again.

### Step 5: Replay a bulk failure in staging

Simulate 200 identical provisioning failures in one run.

Expected: one digest alert, zero pages, and the dedupe key matches across all 200.

## When this applies

- Admins get flooded with near-identical provisioning alerts from one run
- Alert channels get muted, so real distinct failures go unnoticed
- Dedupe exists but identical alerts still arrive separately

## When it doesn't

- Each alert is genuinely distinct and actionable (the volume is the signal)
- Alerts are missing entirely (the opposite problem: fix the alert path)
- The spam comes from a monitoring tool, not the onboarding agent

## Compatibility

PagerDuty, Opsgenie, Slack webhooks, and email alert targets; Okta, Entra, and SCIM provisioning runs as the alert source.

## Variant phrasings

### provisioning run sent 400 slack alerts for one failure

Same fix. The dedupe key must exclude the user id or every user becomes a "distinct" alert.

### alert dedupe not working onboarding bulk import

Check whether the dedupe key includes a timestamp or a random id - either one makes every key unique and defeats dedupe entirely.

## Why it happens

The agent alerts at the point of failure, which is per user, and nobody added an aggregation layer. Dedupe keys that include the user id, timestamp, or a run-unique nonce make every alert "unique" by construction. The result trains admins to ignore the channel, which is worse than no alerts at all because the one distinct failure hides in the flood.

## Edge cases

- A new distinct failure class mid-run deserves its own alert even under the cap
- Retried users that eventually succeed should not appear in the failure digest
- Multi-tenant runs: dedupe per tenant so one tenant's flood does not hide another's failure

## If it still fails

- Capture the dedupe keys of two alerts that should have merged, and the alert config, before changing anything else.
- Reproduce with a small 20-user bulk failure so you are not debugging a crowd.
- Check the alerting provider status page; delivery outages look exactly like dedupe bugs.

## Prevention

- Every alert the agent can send has a documented dedupe key reviewed before launch.
- Load-test alerting with a bulk failure in staging for every new run type.
- Monitor alert volume per run; a spike is itself an alert.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2Hjhzt1I4JFfitV8iHa5Cg