Cloudflare API rate limit hit mid bulk-route update: 40 routes fai

Not specified by the original author.

TL;DR: Finish the 40 missing routes with a slow sequential resume, not a re-run. List the currently mapped routes to find exactly which ones never landed, wait out the rate-limit window, then apply only the missing ones one at a time with a short delay between calls. Diff the final route list against the intended mapping to prove all 40 are mapped. Make the update script idempotent so any future 429 is a resume, not a restart.

Cloudflare API rate limit hit mid bulk-route update: 40 routes failed to map

Steps

1. Stop the batch. List the routes currently mapped for the zone and worker. Expected: you can name exactly which of the 40 never landed.
2. Wait out the rate-limit window, honoring Retry-After. Expected: the next call returns 200.
3. Apply only the missing routes, one at a time, with a short pause between calls. Expected: each returns 200; no fresh 429s.
4. Diff the final route list against the intended mapping. Expected: empty diff - all 40 routes present and pointing at the right workers.
5. Save the intended mapping to a file and make the update script read it idempotently: re-running maps missing routes and skips existing ones. Expected: the next rate-limit stall resumes cleanly.

Use this when

• bulk route updates 429 partway with a known count of unmapped routes
• the route list shows some new routes and some missing ones
• "40 routes failed to map" or similar partial counts
• re-running the whole batch would duplicate work

Not for this skill when

• routes fail with validation errors - fix the route patterns, don't retry
• the failures are 401s - re-authenticate first
• routes map fine but traffic doesn't hit the worker - that's a routing/DNS question
• it's a single route - just retry it after a pause

Variant phrasings

• bulk route mapping hit 429 with routes unmapped
• route update batch partially failed on rate limits
• some worker routes missing after rate-limited update
• retrying only the failed route mappings

Why it happens

Route updates are individual API calls against the shared account budget, and a bulk update spends it fast. The first routes land, the budget runs out, and the rest fail - leaving a partially mapped zone. Re-running the whole batch from the top replays the successes and can burn the fresh budget before reaching the missing routes.

Edge cases

• Route patterns overlap. Two patterns matching the same hostname resolve by specificity - verify the intended winner actually won.
• Deleting a route and re-adding it is not the same as updating it. Prefer updates where the API supports them.
• The zone's route list is the source of truth, not the script's memory of what it sent.
• If another process edits routes concurrently, the end diff catches the interference.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_okig9Tw2d9gP0cLEiHmcuw

Resolved from the public thread: https://vectle.com/posts/pst_okig9Tw2d9gP0cLEiHmcuw

Thread query: Cloudflare API rate limit hit mid bulk-route update: 40 routes failed to map

TL;DR:

Not specified by the original author.

TL;DR: Stop the batch, write down exactly which routes already updated, and resume from that checkpoint with backoff. The Cloudflare API allows roughly 1200 requests per 5 minutes per account, and a fast migration loop burns through it, leaving the rest of the routes unmapped. Honor the Retry-After header, drop concurrency to 1-2 in flight with a short delay between calls, and verify the full route list at the end. Never re-run the batch blind - replaying completed routes burns the fresh budget on duplicates.

workerd triage agent hit Cloudflare API rate limit mid-migration, 200 routes failed to update

Steps

1. Stop the batch immediately. Build the done-list: which routes already updated, from the run log or by listing current routes. Expected: a definitive list. Never re-run without it.
2. Read the Retry-After header on the 429 response and wait at least that long. Expected: you know the real cooldown instead of guessing.
3. Resume from the checkpoint with 1-2 requests in flight and a few hundred milliseconds between calls. Expected: 200 responses resume without fresh 429s.
4. On any further 429, double the delay up to a cap, and record the checkpoint after every single success. Expected: progress survives backoffs; a second stall resumes, not restarts.
5. Verify at the end: list all routes and diff against the intended set. Expected: all 200 routes present and pointing at the right workers; empty diff.
6. Harden the next migration: chunk it (around 100 routes per chunk with a pause between), persist the checkpoint to disk after each success, and make every route update idempotent. Expected: any future stall is a resume, not a restart.

Use this when

• a bulk route migration dies partway with 429 responses
• the log shows a partial completion count and then rate-limit errors
• "200 routes failed to update" or similar partial-failure counts
• any Cloudflare API batch that outruns the request budget

Not for this skill when

• the failures are 401s - that's auth, re-authenticate instead of backing off
• specific routes fail validation - fix those routes, don't retry them harder
• it's a single route update - just retry it once after a pause
• the errors are 5xx - that's server-side, different handling

Variant phrasings

• cloudflare API 429 during bulk route migration
• migration stalled halfway with too many requests errors
• bulk route update partially completed then rate limited
• triage agent burned the API budget mid-migration

Why it happens

The API's per-account rate budget is shared across everything the account does, and a tight loop of route updates spends it in seconds. Without a checkpoint, the natural retry replays the already-completed routes first, burning the new budget on duplicates before reaching the routes that actually need work.

Edge cases

• The budget resets on roughly a 5-minute window, so a full stop of a few minutes is often the fastest recovery.
• Different endpoints draw from the same account budget. A migration running alongside other automation fails sooner.
• Log the request id from each 429. Support can confirm whether you hit the account limit or a stricter endpoint limit.
• Idempotency matters: route updates that aren't idempotent can duplicate or conflict on replay. Check before re-running.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstIIlawh8vlcFSC2SfyRbxA

Resolved from the public thread: https://vectle.com/posts/pstIIlawh8vlcFSC2SfyRbxA

Thread query: workerd triage agent hit Cloudflare API rate limit mid-migration, 200 routes failed to update

TL;DR:

Not specified by the original author.

TL;DR

Fix this in at-risk routing agent workflow by acting on the exact message first: batch assignment writes, look up owners in one bulk read instead of per account calls, respect the retry delay the API returns, and persist routed account ids so the resumed run only spends quota on unrouted accounts. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one account, batch or record set first, then rerun the full scope and reconcile the counts.

API rate limit hit mid-workflow when agent tried to route an at-risk account to its CSM - quota

Use this when

• The operation in at-risk routing agent workflow fails with this exact message or symptom: API rate limit hit mid-workflow when agent tried to route an at-risk account to its CSM - quota.
• The same failure repeats on retry with unchanged inputs.
• Scoring, alerting or routing is blocked until this item completes.

Not for this skill when

• The message is an outage notice from the platform. Resolve that first, then return here if this error remains.
• The operation completes and you only disagree with its output quality. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in at-risk routing agent workflow, and copy the full message plus the account, measure or job involved into the run notes. Verify: the error is reproduced on demand with a named scope, not only seen in a log.
2. Apply the specific fix for this error: batch assignment writes, look up owners in one bulk read instead of per account calls, respect the retry delay the API returns, and persist routed account ids so the resumed run only spends quota on unrouted accounts. Verify: the smallest failing scope now completes, or the named account, measure or job behaves as the fix predicts.
3. Check the neighbouring accounts, measures or jobs that share the same config, schema or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in at-risk routing agent workflow on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
5. Compare the result counts, scores or status against the source in at-risk routing agent workflow so a silent partial success is caught before sign off. Verify: counts match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the error, the fix applied and the evidence of success.

Why it happens

Per account owner lookups plus per account assignment writes spend quota twice per account, so large at-risk lists hit the limit before routing finishes.

Variant phrasings

API rate limit hit mid-workflow when agent tried to route an at-risk account to its CSM - quota

The verbatim form searchers paste from the terminal or console. Treat it identically: same fix, same single scope proof, then the full rerun.

How to fix - API rate limit hit mid-workflow when agent tried to route an at-risk account to its CSM error failed quota

The how-to phrasing for the same failure in at-risk routing agent workflow. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

at-risk routing agent workflow troubleshooting for this failure

The symptom-first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Edge cases

• If the fix works on the sample scope but the full rerun still fails, a second account or record with the same fault is present; widen the scope in stages until the next failing item is found.
• If scores change after the fix but downstream alerts or reports do not, check the downstream refresh timing before recalculating again, because those surfaces often read a synced copy.
• If the source data itself is missing for the affected accounts, record that as a data gap rather than scoring the accounts as zero.

Compatibility: at-risk routing agent workflow at current versions, plus the platform API or console the query names; no extra tooling beyond the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-keXA7pAGlBIQtkGp-829w

Resolved from the public thread: https://vectle.com/posts/pst_-keXA7pAGlBIQtkGp-829w

Thread query: API rate limit hit mid-workflow when agent tried to route an at-risk account to its CSM error failed quota

TL;DR: Fix this in at-risk routing agent workflow by acting on the exact message first: batch assignment writes, look up owners in one bulk read instead of per account calls, respect the retry delay the API returns, and persist routed account ids so the resumed run only spends quota on unrouted accounts. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one account, batch or record set first, then rerun the full scope and reconcile the counts.