## TL;DR
An S3 list call returns at most one thousand keys per page, with a flag saying whether more pages exist and a continuation value for fetching them. Listing code that reads one page and stops silently processes only the oldest thousand files in the prefix, and every later file looks like it never landed. Fix the listing to follow pagination to the end, or use the SDK paginator for the operation, and reconcile the processed set against the full listing so the files already missed get picked up.

```text
agent's s3 prefix listing pagination broke, missed landing files
```

## Use this when
- Files in a landing prefix are present in S3 but the pipeline never processes them.
- The processed files are always the earliest ones in the prefix and the count hovers near a thousand.
- The backlog appeared as the landing zone grew past a small size.

## Not for this skill when
- The files are under a different prefix than the one listed, such as a date folder the job does not expect. That is a prefix bug, not pagination.
- The files fail processing after being listed. Those leave processing errors; missed files leave no trace.

## Steps
1. Count the objects under the prefix with a complete listing from a trusted tool, and compare with how many the job's listing returned in its last run. Note both numbers. Verify: the listing shortfall is measured against the real object count
2. Read the listing code and find the list call: whether it checks the truncated flag, passes a continuation value on the next call, or uses an SDK paginator. Note the exact stopping behaviour. Verify: the pagination gap is located in the code
3. Replace the single call with a paginated loop or the SDK paginator, collecting keys until the listing reports no truncation, and log the page count and total keys per run. Note the new total matches the trusted count. Verify: one listing now returns every key, with the evidence logged
4. Backfill the missed files: diff the full key list against the processed record and submit the missing ones through the normal processing path, watching for files whose moment has passed and need a business decision. Note the missed set submitted. Verify: the historical gap is closed, not just the future one
5. Add a standing reconciliation: each run compares listed keys with processed keys and alerts on a growing unprocessed older than a threshold. Rerun the job and note the counts reconcile. Verify: a future listing bug becomes an alert instead of silent data loss

## Variant phrasings
### s3 ListObjectsV2 pagination continuation token missed objects
The API mechanics phrasing, naming the ListObjectsV2 operation.
### pipeline only processes first 1000 files in s3 prefix
The thousand file symptom search.

Compatibility: Amazon S3 ListObjectsV2 through any AWS SDK, with a one thousand key maximum page size. The pattern applies to any paginated cloud listing API, with different page sizes.

## Why it happens
Pagination bugs are quiet because the first page is real data, correctly processed, so every health signal based on the processed set looks fine. The failure scales with success: the landing zone only crosses a thousand files once the pipeline is actually being used, and from then on the unprocessed tail grows invisibly because listing order means the same oldest page wins every run. Agents and quick scripts are prone to writing the single call version because it works perfectly in every small test, and the API names the continuation mechanism in a way that is easy to overlook.

## Edge cases / pitfalls
- Listing is eventually consistent in edge cases and a brand new file can miss one listing. Reconciliation should allow a grace period before declaring a file missed.
- Very large prefixes make a full in memory key list heavy. Stream the keys and process per page, while still following every page.
- Do not mark files processed at listing time. Mark them when processing succeeds, or the reconciliation cannot tell listed from done.
- Date partitioned prefixes can hide the problem per folder while each folder stays under a page. Reconcile per prefix the job is responsible for.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SLoGo4WQK5wuINZ9tuVouA