agent hit the storage bucket rate limit mid-sweep and the listing failed

TL;DR

Bucket listing is a metadata operation with its own rate limits, and a sweep that lists aggressively across prefixes will be throttled exactly like any abusive client. Pace listing calls, use continuation markers so a throttled listing resumes instead of restarts, prefer bucket inventory exports for large sweeps, and spread listing across time rather than bursting at sweep start.

agent hit the storage bucket rate limit mid-sweep and the listing failed

Use this when

• Agent sweep fails during the listing phase with slow down or rate errors.
• Listing works for small buckets and fails on the largest ones.
• The whole sweep restarts its listing after each throttle.

Not for this skill when

• Restore operations themselves are throttled by the cloud API. Different call family.
• Bucket permissions deny listing outright. Authorisation, not rate.

Steps

1. Identify the listing pattern: page size, parallelism, and whether markers are honoured. Verify: the current listing behaviour is documented
2. Implement marker-based resumption so an interrupted listing continues from the last key. Verify: a killed listing test resumes mid-bucket
3. Add pacing and backoff specific to listing calls. Verify: throttling responses produce slower, successful listing
4. For the largest buckets, switch sweep inventory to the provider inventory export where available. Verify: inventory-driven sweep runs without live listing storms
5. Run the full sweep on the largest bucket. Verify: listing completes and verification proceeds

Variant phrasings

s3 list rate limit sweep inventory export

The inventory phrasing.

bucket listing throttled agent pagination marker

The marker variant.

storage listing slow down backup verification

The slow-down wording.

Compatibility: Agents sweeping S3, Wasabi, B2 and GCS buckets for backup verification, using list APIs and provider inventory reports.

Why it happens

Listing looks read-only and harmless, so agent designers parallelise it freely, but providers rate-limit metadata operations to protect shared control planes. Bursty listing at sweep start is the worst pattern: maximum request rate for zero verification progress. Inventory exports exist precisely to serve bulk knowledge without live listing load.

Edge cases / pitfalls

• A throttled listing is not an empty bucket. Never let rate errors translate into zero objects found.
• Inventory exports are periodic, not live. Use them for sweep planning, live calls for final confirmation.
• Prefixes are not free partitions on every provider. Check the provider rate model before sharding by prefix.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstPD32ZKhkgFqXZQYiaOqaA