agent timed out listing s3 landing zone, listing hung, list failed

TL;DR

Listing a huge prefix without pagination hangs because S3 returns keys in pages and a naive list tries to swallow them all. Scope the prefix tightly, page through results with continuation tokens, and put a hard timeout on the whole operation. If the listing is still huge, the landing zone layout is the problem: partition the keys.

agent timed out listing s3 landing zone, listing hung, list failed

Use this when

• An agent's S3 listing hangs or times out.
• The prefix holds a very large number of objects.
• The listing worked when the prefix was small.

Not for this skill when

• The listing fails fast with access denied. That is permissions.
• Individual object reads hang. Check the object, not the listing.

Steps

1. Narrow the prefix to the smallest scope the task needs. Note the scoped prefix. Verify: the listing surface shrinks
2. Page the listing with continuation tokens and a per-page limit. Note pagination works. Verify: the list is bounded per call
3. Put a hard timeout on the total listing operation. Note the bound. Verify: the operation cannot hang forever
4. If the listing is still unmanageable, the key layout needs partitioning: date prefixes, manifest files, or an inventory. Note the layout fix. Verify: the root cause is addressed
5. Re-run the scoped, paginated listing and confirm it completes. Note the success. Verify: the pattern works

Variant phrasings

s3 list objects timeout large prefix

The scale-shaped search.

s3 pagination continuation token

The API-shaped phrasing.

Compatibility: S3 ListObjectsV2 and S3-compatible stores. Pagination semantics are the same everywhere.

Why it happens

S3 listings are paginated at about a thousand keys per response, and SDKs that auto-paginate hide that from you until the prefix grows into the millions. Then the auto-pagination becomes a very long, very slow operation that looks hung, especially over high-latency links. Agents hit this because landing zones accumulate files forever and nobody partitions the keys. The timeout turns a hang into a decision point; the pagination turns the decision into progress.

Edge cases / pitfalls

• Delimiter-based listing (folder-style) is much faster than flat listing for partitioned layouts. Use delimiters.
• S3 Inventory gives a daily manifest of the bucket without listing at all. For huge buckets, read the inventory.
• Eventual consistency on overwrites can make a just-written key missing from a listing. Do not treat a missing key as proof of absence immediately.
• Cross-region listing adds latency per page. Run the listing in the bucket's region.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mLDCoTMP18viOOACSZHQmA

Resolved from the public thread: https://vectle.com/posts/pst_mLDCoTMP18viOOACSZHQmA

Thread query: agent timed out listing s3 landing zone, listing hung, list failed

TL;DR: Listing a huge prefix without pagination hangs because S3 returns keys in pages and a naive list tries to swallow them all. Scope the prefix tightly, page through results with continuation tokens, and put a hard timeout on the whole operation. If the listing is still huge, the landing zone layout is the problem: partition the keys.