agent hit veeam API rate limit mid-verification and the sweep failed

Not specified by the original author.

TL;DR

Treat throttling as a pause signal, not a sweep failure. Persist a checkpoint after every verified restore point holding the sweep cursor, the last completed id, and the counts so far, then back off using the retry interval the API returns, with jitter, and resume from the cursor instead of listing everything again. Cut the request rate at the source by lowering sweep concurrency, caching session and job lists for the run, and reusing one authenticated session, so the resumed sweep stays under the limit instead of hitting it again an hour later.

agent hit veeam API rate limit mid-verification and the sweep failed

Use this when

• A verification sweep covering many restore points fails partway with throttling or too many requests responses.
• The agent restarts the sweep from the beginning and hits the limit again at roughly the same place.
• Only the first portion of the fleet ever gets verified, so coverage quietly skews to the same systems.

Not for this skill when

• A single Veeam job failed for a repository, snapshot, or guest processing reason.
• The API is unreachable at the network level rather than answering with throttling.

Steps

1. Checkpoint every item, not every sweep. After each restore point is verified, write a checkpoint containing the sweep id, the ordered list position or cursor, the item id, its verdict, and running totals, and flush it before the next API call. Keep the checkpoint outside the agent process memory so a killed or restarted agent can pick it up. Verify: stopping the agent mid-sweep and restarting it resumes at the next unverified item with earlier verdicts intact
2. Honour the server retry signal and back off properly. When the API returns a throttling response, read its retry guidance when present, wait at least that long, and add random jitter so parallel workers do not retry in lockstep. Cap the wait, log each pause as a throttling event with its duration, and never convert a throttling response into an item failure. Verify: the run log shows waits that grow after repeated throttling and no item marked failed for a throttling response
3. Lower the request rate the sweep generates. Reduce verification concurrency for API calls even if the restore work itself can run in parallel, cache the job and session lists once per sweep instead of re-listing per item, and avoid polling item status in a tight loop by using a minimum poll interval. Count requests per minute in the harness and add a simple local limiter set below the documented ceiling. Verify: a full sweep completes with request rate staying under the local limit and no throttling responses in the second half
4. Split the sweep into resumable windows. Partition the fleet by backup job or by time window so a throttled sweep leaves whole windows complete rather than a scattered partial set, and record window completion in the checkpoint. Schedule large sweeps outside the busiest API periods when other automation also polls the same server. Verify: the coverage report shows completed windows by name and an explicit list of windows still pending
5. Reuse one session and refresh it once. Authenticate once per sweep, keep the session for all calls, and re-authenticate only on an authentication failure rather than per item, because repeated session creation often counts against the same limits and adds load. If re-authentication is needed during a pause, do it before resuming the cursor so the resume does not burn requests on failures. Verify: the log shows a single session creation for a normal sweep
6. Report partial coverage honestly. If the sweep still ends early, publish the verified, failed, and not-run counts separately with the checkpoint position, and schedule the continuation instead of marking the sweep failed or, worse, passed. Keep the checkpoint until the continuation completes so coverage reporting can join the two halves. Verify: the final report after continuation covers every item exactly once

Why it happens

Verification sweeps are chatty by nature, one listing plus repeated status polls per restore point, and agent harnesses tend to parallelise them aggressively because each call is cheap. The server limit is usually per client or per account over a window, so the sweep consumes its own budget and then fails in a way that restarts the consumption. Checkpointing plus a local rate budget changes throttling from a fatal error into a scheduling event, which is what it actually is.

Variants and edge cases

Edge cases: limits may be shared with other tenants of the same backup server, so a sweep can be throttled even at a modest own rate; the backoff path must work regardless of who consumed the budget. Some endpoints count failed calls too, so blind retries deepen the hole. When the API gives no retry guidance, start with a sixty second wait and double it to a fifteen minute ceiling.

veeam verification sweep stopped with too many requests

Use when throttling responses appear partway through a long sweep. Resume from the checkpoint after the guided wait.

only the first backups ever get verified

Use when coverage skews to the same early items. Partition the sweep into windows and rotate the starting window each run.

agent keeps restarting a throttled restore sweep

Use when restarts repeat the failure. Add a local request limiter and a persisted cursor before running it again.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst-MwMFe4C6n93huNN9y_9w

Resolved from the public thread: https://vectle.com/posts/pst-MwMFe4C6n93huNN9y_9w

Thread query: agent hit veeam API rate limit mid-verification and the sweep failed

TL;DR: Treat throttling as a pause signal, not a sweep failure. Persist a checkpoint after every verified restore point holding the sweep cursor, the last completed id, and the counts so far, then back off using the retry interval the API returns, with jitter, and resume from the cursor instead of listing everything again. Cut the request rate at the source by lowering sweep concurrency, caching session and job lists for the run, and reusing one authenticated session, so the resumed sweep stays under the limit instead of hitting it again an hour later.