agent's notification failed: pagerduty api returned 429 error

TL;DR

The notifier is sending faster than the API allows. Back off on 429s, batch multiple findings into fewer events, and queue alerts so none are lost.

Error

agent's notification failed: pagerduty api returned 429 error

Steps

1. Read the rate-limit headers to learn the budget. Expected: the real limit.
2. Batch findings into summary events instead of one event per finding. Expected: far fewer calls.
3. Back off exponentially on 429 and retry from a durable queue. Expected: no lost alerts.
4. Deduplicate before sending so repeats do not each cost a call. Expected: less volume.
5. Alert if the queue grows; a growing queue means the budget is structurally too small. Expected: early warning.

When to use

• Notification 429s in agents.
• Designing alert pipelines.

When not to use

• 401/403 errors.
• Alerts never configured (different problem).

Tool compatibility

• PagerDuty Events API; alert queues.

Variant phrasings

pagerduty 429

Batch and back off.

notification rate limited

Same fix.

Why it happens

Each finding becomes an API call; incidents with many findings exceed per-minute budgets instantly.

Edge cases

• Dedup keys must be stable or every retry looks new.
• Queues need persistence; in-memory queues lose alerts on crash.
• Severity routing should survive batching; do not flatten everything to one level.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_akD84tl87Uv1DpuFpKTZXA