agent's S3 list call failed after truncated pagination hid half the objects

TL;DR

A truncated S3 list response is a promise of more data, and an agent that reads only the first page silently verifies half a bucket. Implement the pagination contract strictly: while the response says truncated, request the next page with its continuation marker, accumulate keys, and cross-check the final count against an independent source before the sweep trusts the inventory.

agent's S3 list call failed after truncated pagination hid half the objects

Use this when

• Verification covers only the alphabetically or key-order first slice of a bucket.
• Object counts in sweep reports are round page-size multiples.
• Manual console browsing shows objects the agent never reports.

Not for this skill when

• Listing calls fail with rate errors. Visible failure, different handling.
• Objects are genuinely absent because uploads failed. The listing is honest.

Steps

1. Write a test bucket listing with more objects than one page and trace what the agent collects. Verify: the truncation bug reproduces or is ruled out
2. Fix pagination to loop on the truncated flag using the returned continuation marker until a complete listing is assembled. Verify: the test bucket lists fully
3. Add a guard that refuses to start verification from a listing known to be truncated. Verify: a simulated truncated stop raises an error instead of a partial sweep
4. Cross-check sweep inventory counts against bucket metrics or inventory reports for production buckets. Verify: counts reconcile within reporting lag
5. Re-run the sweep for affected buckets and verify previously hidden objects. Verify: the hidden half is now covered

Variant phrasings

s3 list truncated isTruncated pagination bug

The contract phrasing.

agent missed half bucket objects pagination

The symptom variant.

s3 continuation token loop listing complete

The token wording.

Compatibility: Agents using S3 ListObjects V2 APIs directly or through SDKs with manual pagination, on AWS and compatible stores.

Why it happens

Pagination bugs are silent because the first page is real data, correctly formatted, just incomplete. Everything downstream, verification coverage, gap reports, compliance claims, is then wrong in a way that flatters the system: fewer objects, fewer problems. The truncated flag is the API telling the truth; the agent simply has to keep asking.

Edge cases / pitfalls

• SDK auto-paginators exist for most languages. Prefer them over hand-rolled marker logic.
• A listing that ends exactly on a page boundary still needs the truncation check. Off-by-one lives there.
• Log total listed count per bucket in every sweep. Sudden drops are pagination or deletion events, both worth alarms.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EgVwAAGKXEcfoQKpB4aW0A