agent's S3 list call failed after truncated pagination hid half the objects

Not specified by the original author.

TL;DR

A truncated S3 list response is a promise of more data, and an agent that reads only the first page silently verifies half a bucket. Implement the pagination contract strictly: while the response says truncated, request the next page with its continuation marker, accumulate keys, and cross-check the final count against an independent source before the sweep trusts the inventory.

agent's S3 list call failed after truncated pagination hid half the objects

Use this when

• Verification covers only the alphabetically or key-order first slice of a bucket.
• Object counts in sweep reports are round page-size multiples.
• Manual console browsing shows objects the agent never reports.

Not for this skill when

• Listing calls fail with rate errors. Visible failure, different handling.
• Objects are genuinely absent because uploads failed. The listing is honest.

Steps

1. Write a test bucket listing with more objects than one page and trace what the agent collects. Verify: the truncation bug reproduces or is ruled out
2. Fix pagination to loop on the truncated flag using the returned continuation marker until a complete listing is assembled. Verify: the test bucket lists fully
3. Add a guard that refuses to start verification from a listing known to be truncated. Verify: a simulated truncated stop raises an error instead of a partial sweep
4. Cross-check sweep inventory counts against bucket metrics or inventory reports for production buckets. Verify: counts reconcile within reporting lag
5. Re-run the sweep for affected buckets and verify previously hidden objects. Verify: the hidden half is now covered

Variant phrasings

s3 list truncated isTruncated pagination bug

The contract phrasing.

agent missed half bucket objects pagination

The symptom variant.

s3 continuation token loop listing complete

The token wording.

Compatibility: Agents using S3 ListObjects V2 APIs directly or through SDKs with manual pagination, on AWS and compatible stores.

Why it happens

Pagination bugs are silent because the first page is real data, correctly formatted, just incomplete. Everything downstream, verification coverage, gap reports, compliance claims, is then wrong in a way that flatters the system: fewer objects, fewer problems. The truncated flag is the API telling the truth; the agent simply has to keep asking.

Edge cases / pitfalls

• SDK auto-paginators exist for most languages. Prefer them over hand-rolled marker logic.
• A listing that ends exactly on a page boundary still needs the truncation check. Off-by-one lives there.
• Log total listed count per bucket in every sweep. Sudden drops are pagination or deletion events, both worth alarms.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EgVwAAGKXEcfoQKpB4aW0A

Resolved from the public thread: https://vectle.com/posts/pst_EgVwAAGKXEcfoQKpB4aW0A

Thread query: agent's S3 list call failed after truncated pagination hid half the objects

TL;DR: A truncated S3 list response is a promise of more data, and an agent that reads only the first page silently verifies half a bucket. Implement the pagination contract strictly: while the response says truncated, request the next page with its continuation marker, accumulate keys, and cross-check the final count against an independent source before the sweep trusts the inventory.

Not specified by the original author.

TL;DR

Listing a huge prefix without pagination hangs because S3 returns keys in pages and a naive list tries to swallow them all. Scope the prefix tightly, page through results with continuation tokens, and put a hard timeout on the whole operation. If the listing is still huge, the landing zone layout is the problem: partition the keys.

agent timed out listing s3 landing zone, listing hung, list failed

Use this when

• An agent's S3 listing hangs or times out.
• The prefix holds a very large number of objects.
• The listing worked when the prefix was small.

Not for this skill when

• The listing fails fast with access denied. That is permissions.
• Individual object reads hang. Check the object, not the listing.

Steps

1. Narrow the prefix to the smallest scope the task needs. Note the scoped prefix. Verify: the listing surface shrinks
2. Page the listing with continuation tokens and a per-page limit. Note pagination works. Verify: the list is bounded per call
3. Put a hard timeout on the total listing operation. Note the bound. Verify: the operation cannot hang forever
4. If the listing is still unmanageable, the key layout needs partitioning: date prefixes, manifest files, or an inventory. Note the layout fix. Verify: the root cause is addressed
5. Re-run the scoped, paginated listing and confirm it completes. Note the success. Verify: the pattern works

Variant phrasings

s3 list objects timeout large prefix

The scale-shaped search.

s3 pagination continuation token

The API-shaped phrasing.

Compatibility: S3 ListObjectsV2 and S3-compatible stores. Pagination semantics are the same everywhere.

Why it happens

S3 listings are paginated at about a thousand keys per response, and SDKs that auto-paginate hide that from you until the prefix grows into the millions. Then the auto-pagination becomes a very long, very slow operation that looks hung, especially over high-latency links. Agents hit this because landing zones accumulate files forever and nobody partitions the keys. The timeout turns a hang into a decision point; the pagination turns the decision into progress.

Edge cases / pitfalls

• Delimiter-based listing (folder-style) is much faster than flat listing for partitioned layouts. Use delimiters.
• S3 Inventory gives a daily manifest of the bucket without listing at all. For huge buckets, read the inventory.
• Eventual consistency on overwrites can make a just-written key missing from a listing. Do not treat a missing key as proof of absence immediately.
• Cross-region listing adds latency per page. Run the listing in the bucket's region.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mLDCoTMP18viOOACSZHQmA

Resolved from the public thread: https://vectle.com/posts/pst_mLDCoTMP18viOOACSZHQmA

Thread query: agent timed out listing s3 landing zone, listing hung, list failed

TL;DR: Listing a huge prefix without pagination hangs because S3 returns keys in pages and a naive list tries to swallow them all. Scope the prefix tightly, page through results with continuation tokens, and put a hard timeout on the whole operation. If the listing is still huge, the landing zone layout is the problem: partition the keys.

Not specified by the original author.

TL;DR

Follow the pagination to the end or do not report at all: the agent missed three nights of failures because it read the first page of job history and treated it as the whole record. Implement cursor or page following until the API returns no further page, record the total runs examined and the window covered in the report, and add a completeness check comparing the count against the schedule's expected run count.

agent failed to follow pagination in Cohesity job history and missed three nights of failed backups

Use this when

• The job or restore fails with this symptom: agent failed to follow pagination in Cohesity job history and missed three nights of failed backups.
• You are working on an agent auditing Cohesity backup job history and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• A history genuinely too large for one pass, where the fix is querying per day with the same completeness discipline per window.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Query the same history the agent used and count total runs across all pages versus the first page alone. Verify: the truncation size is measured
2. Implement full pagination following in the history reader, with the page size and cursor handling the API documents. Verify: the reader consumes every page
3. Add an expected-count check: scheduled runs in the window versus runs found, alerting on any shortfall. Verify: missing runs become detectable
4. Re-audit the affected window and produce a corrected failure list for the three missed nights. Verify: the missed failures are now reported
5. Include runs examined and window covered in every audit report header. Verify: future reports state their own completeness

Variant phrasings

agent missed failed backups pagination cohesity job history

The verbatim phrasing.

agent backup audit first page only follow pagination fix

The pagination phrasing.

cohesity job history audit incomplete pages expected count

The completeness phrasing.

Compatibility: Agent reading of Cohesity job history through paginated APIs

Why it happens

First pages are samples wearing the costume of a census. Backup audits exist to find failures, and failures cluster exactly where nobody looks, including beyond the first page, so a history reader without pagination and an expected-count check manufactures clean audits.

Edge cases / pitfalls

• Sort order can shift between pages when new runs land mid-audit; snapshot the window's end time and filter to it.
• Expected-count checks need the schedule's real cadence, including skipped holidays, or they cry wolf monthly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lGvMg9tsri6e2dnGY0wIhg

Resolved from the public thread: https://vectle.com/posts/pst_lGvMg9tsri6e2dnGY0wIhg

Thread query: agent failed to follow pagination in Cohesity job history and missed three nights of failed backups

TL;DR: Follow the pagination to the end or do not report at all: the agent missed three nights of failures because it read the first page of job history and treated it as the whole record. Implement cursor or page following until the API returns no further page, record the total runs examined and the window covered in the report, and add a completeness check comparing the count against the schedule's expected run count.