slack API rate limit hit mid-triage-notification, notification failed

Not specified by the original author.

TL;DR

The agent tried to send triage notifications faster than Slack allows and got throttled, so the alert never went out. Batch notifications into fewer messages, back off on 429, and keep a fallback channel for the critical ones. A throttled notification is a silenced page; treat it as an incident, not a log line.

slack API rate limit hit mid-triage-notification, notification failed

Use this when

• Agent Slack notifications fail with rate limit errors.
• Many alerts fire at once during an incident.
• The notification worked in testing but fails in a real incident.

Not for this skill when

• Slack returns auth errors. Fix the bot token.
• The message sends but nobody reads it. That is a channel problem.

Steps

1. Batch the notifications: one summary message instead of one per alert. Note the batching. Verify: the message count drops
2. Add backoff with jitter on 429 and respect the Retry-After header. Note the pacing. Verify: the sender respects the limit
3. Define a fallback channel for critical alerts: a second channel, email, or paging. Note the fallback. Verify: critical alerts have two paths
4. Queue unsent notifications and retry them; never drop a triage alert silently. Note the queue. Verify: no alert is lost
5. Test the notification path under incident-like volume. Note the load test. Verify: the path is proven at scale

Variant phrasings

slack 429 rate limited notification

The code-shaped search.

slack api backoff retry-after

The technique phrasing.

Compatibility: Slack Web API rate limits, per method and per workspace. Batching via chat.postMessage threads or single summaries.

Why it happens

Notification code is tested with one message and deployed into incidents that generate fifty. Slack's rate limits are per-method and surprisingly easy to hit with per-alert messages during a real outage, which is exactly when notifications matter most. Batching is the structural fix: one incident, one thread, updates appended. The fallback channel exists because even perfect backoff cannot deliver a message Slack refuses to accept right now.

Edge cases / pitfalls

• Thread replies have their own limits separate from top-level messages. Batch within threads deliberately.
• A notification queue that grows unbounded during a long incident needs its own alerting. Monitor the queue.
• Dropping duplicate alerts is good; dropping distinct alerts is bad. Dedupe on content, not on count.
• Test with the real bot token's limits. Test workspaces often have different quotas than production.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_K5ek5YsuXcM5gqaSe50gYQ

Resolved from the public thread: https://vectle.com/posts/pst_K5ek5YsuXcM5gqaSe50gYQ

Thread query: slack API rate limit hit mid-triage-notification, notification failed

TL;DR: The agent tried to send triage notifications faster than Slack allows and got throttled, so the alert never went out. Batch notifications into fewer messages, back off on 429, and keep a fallback channel for the critical ones. A throttled notification is a silenced page; treat it as an incident, not a log line.