agent hit API rate limit mid-playbook-run and the alerts failed to fire

TL;DR

Alerts that travel through the same rate-limited API, or that are dispatched after the step that dies, never leave. The rate limit kills the play and its own alarm. Move alerting to an independent channel, queue alerts before executing risky steps, and make the run's failure handler emit from a path that shares no quota with the work.

agent hit API rate limit mid-playbook-run and the alerts failed to fire

Use this when

• Playbooks failed on rate limits and no alert was received
• The alert mechanism uses the same API or runs after the failing step
• Other alert types from the same system arrive fine

Not for this skill when

• The alert endpoint or channel itself is down
• Alerts fired but into an unmonitored channel

Steps

1. Trace one failed play: where the rate limit hit, where the alert should have been created, and what actually happened. Verify: the alert's death point in the sequence is located
2. Inventory the alert path's dependencies and mark every one shared with the playbook's API quota. Verify: shared failure points are enumerated
3. Move alert dispatch to an independent channel (direct webhook or queue) written before the risky step executes. Verify: a pre-registered alert intent exists even if the play dies instantly
4. Add a watchdog: runs that start without completing or alerting within a window raise an external alert. Verify: silence itself becomes an alertable event
5. Drill it: force a rate-limit failure in staging and confirm the alert arrives. Verify: the decoupled path is proven, not theorised

Variant phrasings

rate limit killed alerts too

A related phrasing for the same underlying problem.

playbook failure no notification

A related phrasing for the same underlying problem.

alert path shares API quota

A related phrasing for the same underlying problem.

Compatibility: Compatibility: agent playbook runners with Slack, PagerDuty, or email alerting independent of CRM or billing APIs, current versions.

Why it happens

An alarm that depends on the failing system is decoration. Rate limits are precisely the failure where the work API is guaranteed unavailable, so the alert path must be independent by construction and verified by drill, not by reading the code.

Edge cases / pitfalls

• Pre-registering alert intent before the step risks false alarms on success paths; resolve the intent on success rather than skipping registration.
• Watchdogs need their own heartbeat, or a dead watchdog repeats the original problem one level up.
• Keep the drill cheap and scheduled. Alert paths rot exactly like the systems they watch.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstQ3S2r2Q8bON2Sw9g9vI-Q