workerd triage agent hit Cloudflare API rate limit mid-migration, 200 routes failed to update

TL;DR: Stop the batch, write down exactly which routes already updated, and resume from that checkpoint with backoff. The Cloudflare API allows roughly 1200 requests per 5 minutes per account, and a fast migration loop burns through it, leaving the rest of the routes unmapped. Honor the Retry-After header, drop concurrency to 1-2 in flight with a short delay between calls, and verify the full route list at the end. Never re-run the batch blind - replaying completed routes burns the fresh budget on duplicates.

workerd triage agent hit Cloudflare API rate limit mid-migration, 200 routes failed to update

Steps

1. Stop the batch immediately. Build the done-list: which routes already updated, from the run log or by listing current routes. Expected: a definitive list. Never re-run without it.
2. Read the Retry-After header on the 429 response and wait at least that long. Expected: you know the real cooldown instead of guessing.
3. Resume from the checkpoint with 1-2 requests in flight and a few hundred milliseconds between calls. Expected: 200 responses resume without fresh 429s.
4. On any further 429, double the delay up to a cap, and record the checkpoint after every single success. Expected: progress survives backoffs; a second stall resumes, not restarts.
5. Verify at the end: list all routes and diff against the intended set. Expected: all 200 routes present and pointing at the right workers; empty diff.
6. Harden the next migration: chunk it (around 100 routes per chunk with a pause between), persist the checkpoint to disk after each success, and make every route update idempotent. Expected: any future stall is a resume, not a restart.

Use this when

• a bulk route migration dies partway with 429 responses
• the log shows a partial completion count and then rate-limit errors
• "200 routes failed to update" or similar partial-failure counts
• any Cloudflare API batch that outruns the request budget

Not for this skill when

• the failures are 401s - that's auth, re-authenticate instead of backing off
• specific routes fail validation - fix those routes, don't retry them harder
• it's a single route update - just retry it once after a pause
• the errors are 5xx - that's server-side, different handling

Variant phrasings

• cloudflare API 429 during bulk route migration
• migration stalled halfway with too many requests errors
• bulk route update partially completed then rate limited
• triage agent burned the API budget mid-migration

Why it happens

The API's per-account rate budget is shared across everything the account does, and a tight loop of route updates spends it in seconds. Without a checkpoint, the natural retry replays the already-completed routes first, burning the new budget on duplicates before reaching the routes that actually need work.

Edge cases

• The budget resets on roughly a 5-minute window, so a full stop of a few minutes is often the fastest recovery.
• Different endpoints draw from the same account budget. A migration running alongside other automation fails sooner.
• Log the request id from each 429. Support can confirm whether you hit the account limit or a stricter endpoint limit.
• Idempotency matters: route updates that aren't idempotent can duplicate or conflict on replay. Check before re-running.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstIIlawh8vlcFSC2SfyRbxA