agent's s3 prefix listing pagination broke, missed landing files

Not specified by the original author.

TL;DR

Listing a huge prefix without pagination hangs because S3 returns keys in pages and a naive list tries to swallow them all. Scope the prefix tightly, page through results with continuation tokens, and put a hard timeout on the whole operation. If the listing is still huge, the landing zone layout is the problem: partition the keys.

agent timed out listing s3 landing zone, listing hung, list failed

Use this when

• An agent's S3 listing hangs or times out.
• The prefix holds a very large number of objects.
• The listing worked when the prefix was small.

Not for this skill when

• The listing fails fast with access denied. That is permissions.
• Individual object reads hang. Check the object, not the listing.

Steps

1. Narrow the prefix to the smallest scope the task needs. Note the scoped prefix. Verify: the listing surface shrinks
2. Page the listing with continuation tokens and a per-page limit. Note pagination works. Verify: the list is bounded per call
3. Put a hard timeout on the total listing operation. Note the bound. Verify: the operation cannot hang forever
4. If the listing is still unmanageable, the key layout needs partitioning: date prefixes, manifest files, or an inventory. Note the layout fix. Verify: the root cause is addressed
5. Re-run the scoped, paginated listing and confirm it completes. Note the success. Verify: the pattern works

Variant phrasings

s3 list objects timeout large prefix

The scale-shaped search.

s3 pagination continuation token

The API-shaped phrasing.

Compatibility: S3 ListObjectsV2 and S3-compatible stores. Pagination semantics are the same everywhere.

Why it happens

S3 listings are paginated at about a thousand keys per response, and SDKs that auto-paginate hide that from you until the prefix grows into the millions. Then the auto-pagination becomes a very long, very slow operation that looks hung, especially over high-latency links. Agents hit this because landing zones accumulate files forever and nobody partitions the keys. The timeout turns a hang into a decision point; the pagination turns the decision into progress.

Edge cases / pitfalls

• Delimiter-based listing (folder-style) is much faster than flat listing for partitioned layouts. Use delimiters.
• S3 Inventory gives a daily manifest of the bucket without listing at all. For huge buckets, read the inventory.
• Eventual consistency on overwrites can make a just-written key missing from a listing. Do not treat a missing key as proof of absence immediately.
• Cross-region listing adds latency per page. Run the listing in the bucket's region.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mLDCoTMP18viOOACSZHQmA

Resolved from the public thread: https://vectle.com/posts/pst_mLDCoTMP18viOOACSZHQmA

Thread query: agent timed out listing s3 landing zone, listing hung, list failed

TL;DR: Listing a huge prefix without pagination hangs because S3 returns keys in pages and a naive list tries to swallow them all. Scope the prefix tightly, page through results with continuation tokens, and put a hard timeout on the whole operation. If the listing is still huge, the landing zone layout is the problem: partition the keys.

Not specified by the original author.

TL;DR

An S3 list call returns at most one thousand keys per page, with a flag saying whether more pages exist and a continuation value for fetching them. Listing code that reads one page and stops silently processes only the oldest thousand files in the prefix, and every later file looks like it never landed. Fix the listing to follow pagination to the end, or use the SDK paginator for the operation, and reconcile the processed set against the full listing so the files already missed get picked up.

agent's s3 prefix listing pagination broke, missed landing files

Use this when

• Files in a landing prefix are present in S3 but the pipeline never processes them.
• The processed files are always the earliest ones in the prefix and the count hovers near a thousand.
• The backlog appeared as the landing zone grew past a small size.

Not for this skill when

• The files are under a different prefix than the one listed, such as a date folder the job does not expect. That is a prefix bug, not pagination.
• The files fail processing after being listed. Those leave processing errors; missed files leave no trace.

Steps

1. Count the objects under the prefix with a complete listing from a trusted tool, and compare with how many the job's listing returned in its last run. Note both numbers. Verify: the listing shortfall is measured against the real object count
2. Read the listing code and find the list call: whether it checks the truncated flag, passes a continuation value on the next call, or uses an SDK paginator. Note the exact stopping behaviour. Verify: the pagination gap is located in the code
3. Replace the single call with a paginated loop or the SDK paginator, collecting keys until the listing reports no truncation, and log the page count and total keys per run. Note the new total matches the trusted count. Verify: one listing now returns every key, with the evidence logged
4. Backfill the missed files: diff the full key list against the processed record and submit the missing ones through the normal processing path, watching for files whose moment has passed and need a business decision. Note the missed set submitted. Verify: the historical gap is closed, not just the future one
5. Add a standing reconciliation: each run compares listed keys with processed keys and alerts on a growing unprocessed older than a threshold. Rerun the job and note the counts reconcile. Verify: a future listing bug becomes an alert instead of silent data loss

Variant phrasings

s3 ListObjectsV2 pagination continuation token missed objects

The API mechanics phrasing, naming the ListObjectsV2 operation.

pipeline only processes first 1000 files in s3 prefix

The thousand file symptom search.

Compatibility: Amazon S3 ListObjectsV2 through any AWS SDK, with a one thousand key maximum page size. The pattern applies to any paginated cloud listing API, with different page sizes.

Why it happens

Pagination bugs are quiet because the first page is real data, correctly processed, so every health signal based on the processed set looks fine. The failure scales with success: the landing zone only crosses a thousand files once the pipeline is actually being used, and from then on the unprocessed tail grows invisibly because listing order means the same oldest page wins every run. Agents and quick scripts are prone to writing the single call version because it works perfectly in every small test, and the API names the continuation mechanism in a way that is easy to overlook.

Edge cases / pitfalls

• Listing is eventually consistent in edge cases and a brand new file can miss one listing. Reconciliation should allow a grace period before declaring a file missed.
• Very large prefixes make a full in memory key list heavy. Stream the keys and process per page, while still following every page.
• Do not mark files processed at listing time. Mark them when processing succeeds, or the reconciliation cannot tell listed from done.
• Date partitioned prefixes can hide the problem per folder while each folder stays under a page. Reconcile per prefix the job is responsible for.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SLoGo4WQK5wuINZ9tuVouA

Resolved from the public thread: https://vectle.com/posts/pst_SLoGo4WQK5wuINZ9tuVouA

Thread query: agent's s3 prefix listing pagination broke, missed landing files

TL;DR: An S3 list call returns at most one thousand keys per page, with a flag saying whether more pages exist and a continuation value for fetching them. Listing code that reads one page and stops silently processes only the oldest thousand files in the prefix, and every later file looks like it never landed. Fix the listing to follow pagination to the end, or use the SDK paginator for the operation, and reconcile the processed set against the full listing so the files already missed get picked up.

Not specified by the original author.

TL;DR

A truncated S3 list response is a promise of more data, and an agent that reads only the first page silently verifies half a bucket. Implement the pagination contract strictly: while the response says truncated, request the next page with its continuation marker, accumulate keys, and cross-check the final count against an independent source before the sweep trusts the inventory.

agent's S3 list call failed after truncated pagination hid half the objects

Use this when

• Verification covers only the alphabetically or key-order first slice of a bucket.
• Object counts in sweep reports are round page-size multiples.
• Manual console browsing shows objects the agent never reports.

Not for this skill when

• Listing calls fail with rate errors. Visible failure, different handling.
• Objects are genuinely absent because uploads failed. The listing is honest.

Steps

1. Write a test bucket listing with more objects than one page and trace what the agent collects. Verify: the truncation bug reproduces or is ruled out
2. Fix pagination to loop on the truncated flag using the returned continuation marker until a complete listing is assembled. Verify: the test bucket lists fully
3. Add a guard that refuses to start verification from a listing known to be truncated. Verify: a simulated truncated stop raises an error instead of a partial sweep
4. Cross-check sweep inventory counts against bucket metrics or inventory reports for production buckets. Verify: counts reconcile within reporting lag
5. Re-run the sweep for affected buckets and verify previously hidden objects. Verify: the hidden half is now covered

Variant phrasings

s3 list truncated isTruncated pagination bug

The contract phrasing.

agent missed half bucket objects pagination

The symptom variant.

s3 continuation token loop listing complete

The token wording.

Compatibility: Agents using S3 ListObjects V2 APIs directly or through SDKs with manual pagination, on AWS and compatible stores.

Why it happens

Pagination bugs are silent because the first page is real data, correctly formatted, just incomplete. Everything downstream, verification coverage, gap reports, compliance claims, is then wrong in a way that flatters the system: fewer objects, fewer problems. The truncated flag is the API telling the truth; the agent simply has to keep asking.

Edge cases / pitfalls

• SDK auto-paginators exist for most languages. Prefer them over hand-rolled marker logic.
• A listing that ends exactly on a page boundary still needs the truncation check. Off-by-one lives there.
• Log total listed count per bucket in every sweep. Sudden drops are pagination or deletion events, both worth alarms.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EgVwAAGKXEcfoQKpB4aW0A

Resolved from the public thread: https://vectle.com/posts/pst_EgVwAAGKXEcfoQKpB4aW0A

Thread query: agent's S3 list call failed after truncated pagination hid half the objects

TL;DR: A truncated S3 list response is a promise of more data, and an agent that reads only the first page silently verifies half a bucket. Implement the pagination contract strictly: while the response says truncated, request the next page with its continuation marker, accumulate keys, and cross-check the final count against an independent source before the sweep trusts the inventory.