agent's browser automation hit a rate limit mid-suite: how to resume

TL;DR

When the suite hits a rate limit (429s), stop hammering: pause new requests, honor the Retry-After header, checkpoint which tests finished, then resume from the checkpoint at a throttled pace. Retrying into a rate limit just extends the ban.

Error

GET https://example.com/api/search ... 429 Too Many Requests
Retry-After: 120
test_results_page_03 FAILED: expected 200, got 429
test_results_page_04 FAILED: expected 200, got 429

The suite burned through the site's quota mid-run and every subsequent test failed on 429s.

Steps

1. Detect the limit early: fail the run fast on the first 429 instead of letting 50 more tests fail. Grep logs for 429 or check response status in your request helper. Expected: one clear rate-limit signal, not a wall of identical failures.
2. Read the Retry-After header (or the API's documented quota window) and wait it out; use exponential backoff with jitter for retries, starting at the header value. Expected: the next request goes out after the window, not into it.
3. Checkpoint progress: record which tests completed before the 429s started (reporter output or a simple done-list file). Expected: a resume list, so finished tests are not re-run against the quota.
4. Resume at a throttled pace: cut concurrency (for example 4 workers down to 1), add a fixed delay between requests (for example 2 seconds), and spread the remaining tests over the quota window. Expected: the resumed run stays under the limit.
5. For the future: give the agent a per-suite request budget and a pre-run quota check. Expected: the agent estimates requests before running and warns if the suite needs more than the quota allows.

When to use

• Browser automation or API-driven tests start returning 429 mid-suite.
• An agent's suite hammers a staging or third-party endpoint with no throttling.
• You need to salvage a partially completed run without re-spending quota.

When not to use

• The 429s come from your own WAF or bot protection blocking automation (that is access policy, not quota; throttling will not help).
• A single test gets 429 while others pass (probably that test's bug, such as a retry loop; fix the test).
• The rate limit is per key and the key is shared with production traffic (rotate to a dedicated test key instead).

Tool compatibility

• Playwright, Puppeteer, Cypress, or any HTTP-driven browser automation hitting real endpoints.
• Works with any site that returns standard 429 plus Retry-After.

Variant phrasings

Suite got IP-blocked after too many requests

A block is a rate limit with a longer memory. Same resume playbook, plus rotate the egress IP or wait out the block window before resuming.

Rate limit hit only on image or asset requests

Throttle or mock static assets (route them to local fixtures). The suite rarely needs real images to test behavior.

Agent retried 429s aggressively and made it worse

Classic. The fix is the backoff policy plus a circuit breaker: after 3 consecutive 429s, pause the whole run instead of retrying per test.

Why it happens

Test suites are the most aggressive clients a site ever sees: dozens of parallel workers hitting the same endpoints with no human pauses. Agents do not feel social pressure to slow down, so they blow through quotas that a human tester would never touch. The site defends itself with 429s, and every retry during the window just resets the clock.

Edge cases

• Retry-After lies (some servers send 0 or omit it): fall back to exponential backoff with jitter, capped at 5 minutes.
• Quota windows that reset hourly: schedule the resume after the reset, not after a fixed sleep.
• Tests that must run in order with shared server-side state: throttling changes timing, which can expose order-dependent flakes; keep the delay constant so timing stays deterministic-ish.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RhA02RfW29EQczLI0N-U1g