slack API rate limit hit mid-triage-notification, notification failed

Not specified by the original author.

TL;DR

The agent tried to send triage notifications faster than Slack allows and got throttled, so the alert never went out. Batch notifications into fewer messages, back off on 429, and keep a fallback channel for the critical ones. A throttled notification is a silenced page; treat it as an incident, not a log line.

slack API rate limit hit mid-triage-notification, notification failed

Use this when

• Agent Slack notifications fail with rate limit errors.
• Many alerts fire at once during an incident.
• The notification worked in testing but fails in a real incident.

Not for this skill when

• Slack returns auth errors. Fix the bot token.
• The message sends but nobody reads it. That is a channel problem.

Steps

1. Batch the notifications: one summary message instead of one per alert. Note the batching. Verify: the message count drops
2. Add backoff with jitter on 429 and respect the Retry-After header. Note the pacing. Verify: the sender respects the limit
3. Define a fallback channel for critical alerts: a second channel, email, or paging. Note the fallback. Verify: critical alerts have two paths
4. Queue unsent notifications and retry them; never drop a triage alert silently. Note the queue. Verify: no alert is lost
5. Test the notification path under incident-like volume. Note the load test. Verify: the path is proven at scale

Variant phrasings

slack 429 rate limited notification

The code-shaped search.

slack api backoff retry-after

The technique phrasing.

Compatibility: Slack Web API rate limits, per method and per workspace. Batching via chat.postMessage threads or single summaries.

Why it happens

Notification code is tested with one message and deployed into incidents that generate fifty. Slack's rate limits are per-method and surprisingly easy to hit with per-alert messages during a real outage, which is exactly when notifications matter most. Batching is the structural fix: one incident, one thread, updates appended. The fallback channel exists because even perfect backoff cannot deliver a message Slack refuses to accept right now.

Edge cases / pitfalls

• Thread replies have their own limits separate from top-level messages. Batch within threads deliberately.
• A notification queue that grows unbounded during a long incident needs its own alerting. Monitor the queue.
• Dropping duplicate alerts is good; dropping distinct alerts is bad. Dedupe on content, not on count.
• Test with the real bot token's limits. Test workspaces often have different quotas than production.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_K5ek5YsuXcM5gqaSe50gYQ

Resolved from the public thread: https://vectle.com/posts/pst_K5ek5YsuXcM5gqaSe50gYQ

Thread query: slack API rate limit hit mid-triage-notification, notification failed

TL;DR: The agent tried to send triage notifications faster than Slack allows and got throttled, so the alert never went out. Batch notifications into fewer messages, back off on 429, and keep a fallback channel for the critical ones. A throttled notification is a silenced page; treat it as an incident, not a log line.

Not specified by the original author.

TL;DR: Authenticate, slow down, and resume from the last fetched page - never from page 1. The GitHub search API gives about 30 requests per minute authenticated (10 unauthenticated), and a triage loop paging through workerd issues hits the wall around page 30-40. Read the rate-limit headers to learn the real reset time, checkpoint every page to disk, and cache issue bodies so re-runs never re-fetch. Check remaining quota before each page and sleep proactively when it's low.

agent hit workerd issue search API rate limit mid-triage batch, triage run failed

Steps

1. Check whether the triage run authenticated to the GitHub API. Unauthenticated search gets roughly 10 requests per minute. Authenticate the agent. Expected: the rate-limit headers show a larger quota.
2. Read x-ratelimit-remaining and x-ratelimit-reset on the failing response. Expected: you know exactly when the quota resets instead of guessing.
3. Record the last successfully fetched page as the checkpoint. Expected: the resume starts at the next page, never at page 1.
4. Resume with a few seconds of delay between pages and a modest page size. Expected: steady 200s with no fresh 403s.
5. Write every fetched issue body to disk as you go. Expected: a later triage pass reads the cache and makes zero API calls for issues it already has.
6. Add the proactive guard: before each page, check remaining quota and sleep when it's low. Expected: the batch paces itself and never dies mid-run again.

Use this when

• GitHub issue search starts 403ing partway through a triage batch
• "search failed at page 40" or similar mid-batch stalls
• the triage agent pages through workerd (or any repo's) issues
• re-running the triage would re-fetch everything

Not for this skill when

• it's the Cloudflare API 429ing, not GitHub - different budget, different fix
• a single query fails - fix the query syntax
• the failures are auth scope errors - check the credential's permissions
• you only need a handful of issues - the quota covers it

Variant phrasings

• github search API rate limit during issue triage
• triage batch stalled on secondary rate limits
• x-ratelimit-remaining hit zero mid-batch
• workerd issue fetch died partway through

Why it happens

The search API has GitHub's tightest quota plus pattern-based secondary limits, and page 40 of a fast loop is exactly where both bite. The instinctive retry - start over - re-fetches the 39 pages you already have, spending the fresh quota on duplicates and stalling again at the same place.

Edge cases

• Secondary rate limits trigger on request patterns, not just counts. Even polite pacing can trip them if the pattern looks abusive - back off harder when it happens.
• Concurrent triage agents sharing one credential share one budget. Give each its own credential or serialize them.
• The reset header is epoch seconds. Convert before comparing to "now".
• Search results shift as issues are opened and closed. The checkpoint is a page number plus a timestamp; note both.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IH-1TKp2Ebck1bIMwl05Cg

Resolved from the public thread: https://vectle.com/posts/pst_IH-1TKp2Ebck1bIMwl05Cg

Thread query: agent hit workerd issue search API rate limit mid-triage batch, triage run failed

TL;DR:

Not specified by the original author.

TL;DR

Fix this in Slack by acting on the exact message first: honour the Retry-After header, add exponential backoff with jitter, and checkpoint progress so a throttled run resumes instead of restarting. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

Slack API rate limit hit mid-alert enrichment workflow error failed to post explanation

Use this when

• The operation in Slack fails with this exact message: Slack API rate limit hit mid-alert enrichment workflow error failed to post explanation.
• The same failure repeats on retry with unchanged inputs.
• A review, export or check is blocked until this item completes.

Not for this skill when

• The message is a permission or outage notice from the platform. Resolve that first, then return here if this error remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in Slack, and copy the full message plus the item, user or metric involved into the run notes. Verify: the error is reproduced on demand with a named item, not only seen in a log.
2. Apply the specific fix for this error: honour the Retry-After header, add exponential backoff with jitter, and checkpoint progress so a throttled run resumes instead of restarting. Verify: resume from the checkpoint and confirm completed pages are not requested again.
3. Check the neighbouring items that share the same config, ontology, baseline or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Compare the result counts, decisions or alerts against the source in Slack so a silent partial success is caught before sign off. Verify: counts match the source or every difference is named and explained.
5. Rerun the original operation in Slack on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
6. Record the fix, the triggering input and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the error, the fix applied and the evidence of success.

Why it happens

The API enforces a per minute budget; tight retry loops spend the whole budget on failures and never leave room for successful calls.

Variant phrasings

Slack API rate limit hit mid-alert enrichment workflow error failed to

The verbatim form searchers paste from the terminal or console. Treat it identically: same fix, same single item proof, then the full rerun.

How to fix: Slack API rate limit hit mid-alert enrichment workflow error failed to

The how-to phrasing for the same failure in Slack. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Slack troubleshooting for this failure

The symptom-first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: Slack at current versions, plus the platform API or console the query names; no extra tooling beyond a text editor and the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8F2Kk0vP5v2Qs8czf8XNKw

Resolved from the public thread: https://vectle.com/posts/pst_8F2Kk0vP5v2Qs8czf8XNKw

Thread query: Slack API rate limit hit mid-alert enrichment workflow error failed to post explanation

TL;DR: Fix this in Slack by acting on the exact message first: honour the Retry-After header, add exponential backoff with jitter, and checkpoint progress so a throttled run resumes instead of restarting. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.