agent broke out of the pagination loop on the first empty page - the API legitimately returns empty pages mid-list

TL;DR

An empty page is not the end of the list on providers that return sparse pages. Keep paginating until the cursor itself says stop - null, missing, or repeating - and treat empty pages as "nothing here, keep going." The data after the gap is real and the client was leaving it behind.

The query

agent broke out of the pagination loop on the first empty page  -  the API legitimately returns empty pages mid-list

Steps

1. Confirm empty pages are legitimate on this provider

Check the provider's pagination docs for sparse or empty pages, or observe them directly: paginate manually past an empty page and see whether later pages contain items. If they do, empty pages are normal.

Expected: a yes from docs or direct observation that empty mid-list pages happen.

2. Move the termination condition to the cursor

Rewrite the loop to break only when the end-of-list signal fires: the next cursor is null, missing, or repeats a seen cursor. Remove any if not items: break logic. The loop condition should mention the cursor, not the item count.

Expected: a loop that pages through empty pages without stopping.

3. Keep collecting across the gaps

Make sure the accumulator appends whatever each page returns (possibly nothing) and continues. Do not reset state or short-circuit on an empty page.

Expected: the collected list includes items from pages after the empty ones.

4. Retain the safety backstops

Empty-tolerant loops still need the seen-cursor set and the maximum page count. Tolerance for empty pages plus a missing backstop is how a different infinite loop starts.

Expected: the loop ends on cursor exhaustion or the cap, never on emptiness.

5. Verify with a list known to have gaps

Run the fixed client against a list with known sparse regions and confirm the collected items match the expected full set, including items positioned after empty pages.

Expected: complete item set, loop exited cleanly via the cursor signal.

Use this when

• Fetches stop early with no error and the tail of the list is missing
• The generated code breaks on an empty data array
• The provider documents sparse pages, or you have observed items after an empty page
• The missing items cluster at the end of the list

Not for this skill when

• The provider signals the end WITH an empty page (then breaking is correct)
• The list is empty because of a filter or permission (check the query, not the loop)
• The cursor logic itself is broken (fix the cursor first)
• The endpoint genuinely returns all items on page one (no pagination needed)

Variant phrasings

loop stopped early but the provider has more data

Same fix. "More data exists" plus "loop stopped on empty" is the complete diagnosis.

tail of the list never arrived

The items after the first gap are the ones that go missing. Step 3's keep-going accumulator recovers them.

Why it happens

"Stop when the page is empty" is the intuitive termination rule and it works on providers with dense pages, so the agent generated it without checking. This provider's pages are sparse - filters, sharding, or soft deletes leave genuine gaps. The agent's mental model said empty means done; the provider's model says empty means keep going. One observed gap would have corrected the model, but the happy-path test data had no gaps.

Edge cases

• Provider signals the end with an empty page AND a null cursor: breaking on empty is accidentally correct there. Still prefer the cursor condition - it stays correct if the provider changes the convention.
• Long runs of empty pages: fine, the cursor still advances. The page-count backstop covers pathological cases.
• Empty first page: the client should make its requests and exit via the cursor, returning an empty collection - not crash on "no data."
• Distinguishing "empty page, more coming" from "empty page, done": only the cursor distinguishes them. That is the whole point of step 2.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_N1zLElWwQ68Dcq7tNWGXug