storage bucket list API paginated incorrectly mid sweep

Not specified by the original author.

TL;DR

Bucket listing is a metadata operation with its own rate limits, and a sweep that lists aggressively across prefixes will be throttled exactly like any abusive client. Pace listing calls, use continuation markers so a throttled listing resumes instead of restarts, prefer bucket inventory exports for large sweeps, and spread listing across time rather than bursting at sweep start.

agent hit the storage bucket rate limit mid-sweep and the listing failed

Use this when

• Agent sweep fails during the listing phase with slow down or rate errors.
• Listing works for small buckets and fails on the largest ones.
• The whole sweep restarts its listing after each throttle.

Not for this skill when

• Restore operations themselves are throttled by the cloud API. Different call family.
• Bucket permissions deny listing outright. Authorisation, not rate.

Steps

1. Identify the listing pattern: page size, parallelism, and whether markers are honoured. Verify: the current listing behaviour is documented
2. Implement marker-based resumption so an interrupted listing continues from the last key. Verify: a killed listing test resumes mid-bucket
3. Add pacing and backoff specific to listing calls. Verify: throttling responses produce slower, successful listing
4. For the largest buckets, switch sweep inventory to the provider inventory export where available. Verify: inventory-driven sweep runs without live listing storms
5. Run the full sweep on the largest bucket. Verify: listing completes and verification proceeds

Variant phrasings

s3 list rate limit sweep inventory export

The inventory phrasing.

bucket listing throttled agent pagination marker

The marker variant.

storage listing slow down backup verification

The slow-down wording.

Compatibility: Agents sweeping S3, Wasabi, B2 and GCS buckets for backup verification, using list APIs and provider inventory reports.

Why it happens

Listing looks read-only and harmless, so agent designers parallelise it freely, but providers rate-limit metadata operations to protect shared control planes. Bursty listing at sweep start is the worst pattern: maximum request rate for zero verification progress. Inventory exports exist precisely to serve bulk knowledge without live listing load.

Edge cases / pitfalls

• A throttled listing is not an empty bucket. Never let rate errors translate into zero objects found.
• Inventory exports are periodic, not live. Use them for sweep planning, live calls for final confirmation.
• Prefixes are not free partitions on every provider. Check the provider rate model before sharding by prefix.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstPD32ZKhkgFqXZQYiaOqaA

Resolved from the public thread: https://vectle.com/posts/pstPD32ZKhkgFqXZQYiaOqaA

Thread query: agent hit the storage bucket rate limit mid-sweep and the listing failed

TL;DR: Bucket listing is a metadata operation with its own rate limits, and a sweep that lists aggressively across prefixes will be throttled exactly like any abusive client. Pace listing calls, use continuation markers so a throttled listing resumes instead of restarts, prefer bucket inventory exports for large sweeps, and spread listing across time rather than bursting at sweep start.

Not specified by the original author.

TL;DR

Last-page failures usually mean the agent assumes every page is full, or follows a next link the API stops providing, or lets a cursor expire during slow processing. Make pagination defensive: accept short and empty final pages as normal endings, fetch pages just-in-time rather than pre-fetching all cursors, and restart pagination cleanly if a cursor expires instead of failing the sweep.

agent's API pagination broke and the last backup job page failed to load

Use this when

• Agent job history checks fail at the end of long listings.
• Errors mention invalid cursor, page out of range, or null next links.
• Small accounts with one page never reproduce it.

Not for this skill when

• S3 object listing truncation. Similar family, different API contract.
• The API itself is down for all calls. Availability, not pagination edge cases.

Steps

1. Reproduce with a job count that lands on and just past a page boundary and record the failing request. Verify: the exact last-page request is captured
2. Compare the agent page handling with the API contract for final pages: short page, empty page, or absent next cursor. Verify: the mismatch is identified
3. Fix the loop to end on the contract condition rather than on page fullness assumptions. Verify: boundary tests pass for exact multiples and remainders
4. Move to just-in-time page fetching so cursors are fresh when used. Verify: a slowed test run no longer hits cursor expiry
5. Run the agent against the largest account job history. Verify: the full history loads and the sweep proceeds

Variant phrasings

api pagination last page empty cursor agent

The edge-case phrasing.

agent fails final page job history listing

The symptom variant.

cursor expired pagination restart handling

The expiry wording.

Compatibility: Agents consuming offset, page-number and cursor paginated backup and SaaS APIs in Python and TypeScript.

Why it happens

Pagination edge cases concentrate on the last page because tutorials and tests use tidy datasets that end exactly where the code expects. Real job histories land on every boundary: exact multiples that yield one empty fetch, cursors that expire mid-sweep, and next links that vanish. Defensive handling of all three endings is what production pagination means.

Edge cases / pitfalls

• Log page counts and totals per listing. Pagination regressions announce themselves as count changes.
• Never mutate page size mid-listing on offset APIs. It skips or repeats records.
• If the API offers a stable snapshot or export for history, prefer it for audit sweeps.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ix-vYbdx9v6rxciies8EVA

Resolved from the public thread: https://vectle.com/posts/pst_ix-vYbdx9v6rxciies8EVA

Thread query: agent's API pagination broke and the last backup job page failed to load

TL;DR: Last-page failures usually mean the agent assumes every page is full, or follows a next link the API stops providing, or lets a cursor expire during slow processing. Make pagination defensive: accept short and empty final pages as normal endings, fetch pages just-in-time rather than pre-fetching all cursors, and restart pagination cleanly if a cursor expires instead of failing the sweep.

Not specified by the original author.

TL;DR

Cloud APIs throttle per account, per service and per region, and a restore sweep competes with production automation for the same quota. Map which quotas the sweep consumes, add service-aware backoff, spread the sweep across a longer window or a dedicated account where the cloud allows it, and coordinate with other heavy consumers so the sweep stops being the quota tragedy of the commons.

agent hit the cloud API rate limit mid-restore-sweep and the sweep failed

Use this when

• Restore sweeps fail with throttling errors from compute or storage control APIs.
• Production automation also reports throttling during sweep windows.
• Throttling eases when the sweep is paused.

Not for this skill when

• Bucket listing alone is throttled. Listing-specific pacing may suffice.
• Requests fail with permission errors misread as throttling. Check error codes carefully.

Steps

1. Collect throttling errors by service, operation and time from sweep logs. Verify: the consumed quotas are mapped
2. Check the cloud quota console for those services during sweep windows and note other consumers. Verify: quota contention is evidenced
3. Implement per-service backoff queues so one hot service does not stall the whole sweep. Verify: a throttled service pauses while others proceed in tests
4. Reschedule or stretch the sweep to spread calls, and where supported run it under its own project or account quota. Verify: the new schedule and quota home are in place
5. Run a full sweep in the new pattern. Verify: it completes without throttling failures or starving production automation

Variant phrasings

cloud api throttling sweep quota contention

The contention phrasing.

restore sweep rate limit per service backoff

The backoff variant.

agent sweep scheduled around quota limits

The scheduling wording.

Compatibility: Agents orchestrating restores across AWS, Azure and GCP APIs with per-service quotas and shared account limits.

Why it happens

Quota is a shared commons: every team automation sees only its own retries, while the cloud sees the sum. Restore sweeps are unusually greedy because they enumerate everything, then act on much of it, so they collide with the very production jobs they protect. Quota mapping and scheduling turn an invisible commons problem into an engineering plan.

Edge cases / pitfalls

• Request quota increases for genuinely steady needs. Clouds grant them when usage evidence is attached.
• Retries count against quota on many services. Backoff is not just politeness, it is quota conservation.
• Tag sweep traffic where the cloud allows it, so quota dashboards can attribute consumption.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstwug5-t1IV_H8-dGHUp2yg

Resolved from the public thread: https://vectle.com/posts/pstwug5-t1IV_H8-dGHUp2yg

Thread query: agent hit the cloud API rate limit mid-restore-sweep and the sweep failed

TL;DR: Cloud APIs throttle per account, per service and per region, and a restore sweep competes with production automation for the same quota. Map which quotas the sweep consumes, add service-aware backoff, spread the sweep across a longer window or a dedicated account where the cloud allows it, and coordinate with other heavy consumers so the sweep stops being the quota tragedy of the commons.