agent failed mid-rollout of patch wave after rate limit hit

TL;DR

A 429 mid-rollout is a pacing signal, not a failure: the agent should honor Retry-After, back off, checkpoint its position, and resume, rather than crashing the wave or retrying instantly and deepening the throttle. Implement a shared rate limiter for the API client, persist per-target completion as you go, and treat 429 as a pause state in the wave tracker.

agent failed mid-rollout of patch wave after rate limit hit

Use this when

• Wave logs end abruptly after 429 or Too Many Requests responses.
• The failure happens at a consistent point, such as after about N API calls.
• Restarting the wave redoes work on hosts already patched.

Not for this skill when

• The API returns 401 or 403. That is auth, not rate; retrying will not help.
• The daily quota is genuinely exhausted. The correct action is to schedule the remainder, not to pace it.

Steps

1. Find where the 429 surfaces and whether the response carries a Retry-After header. Verify: the client logs the limit, remaining, and reset values when present
2. Add a token-bucket or fixed-window limiter in the API client sized below the documented limit, shared across all parallel workers. Verify: parallel test calls stay under the limit
3. Make each completed target write its result to the wave tracker immediately (per item, not at wave end). Verify: killing the agent mid-wave loses no completed results
4. On 429, set the wave to paused-rate-limit state, sleep for Retry-After (or exponential backoff), and resume from the tracker position. Verify: a throttled test resumes at the first uncompleted target
5. Report throttling in the wave summary: count of 429s, total pause time, and where the limit sits relative to wave size. Verify: the report shows whether the wave size itself needs splitting

Variant phrasings

agent 429 rate limit mid rollout resume checkpoint

The fix-pattern phrasing.

honor retry-after header patch api client

The client-behavior phrasing.

parallel workers shared rate limiter fleet api

The concurrency phrasing.

Compatibility: Compatibility: Any agent calling rate-limited REST APIs (patch systems, CMDB, cloud control planes); limits and headers vary by vendor.

Why it happens

Agents fan out work in parallel, so their aggregate request rate is easy to underestimate: ten workers polling status can exceed a limit that one sequential script never approached. Without checkpointing, a throttle at 80 percent completion throws away the whole wave, and the restart doubles the load that caused the throttle, making the next failure earlier.

Edge cases / pitfalls

• Status polling is usually the biggest call consumer; lengthen poll intervals and prefer webhooks or batch status endpoints where they exist.
• A limiter per worker is not a limiter; the budget must be shared or divided explicitly across workers.
• If waves routinely need more calls than the limit allows per window, negotiate the limit or pre-fetch data in bulk; pacing alone cannot create capacity.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lOkOaAi0KUFh65966utT7Q