generated client paginated with page_size=100 but the API silently caps at 50 - half the records never arrived

TL;DR

The provider is allowed to ignore your page size, and some do it silently. After the first request, read back how many items actually came back and use that as the real page size - or better, do not depend on page size at all and just follow the cursor until the list ends. The missing half of the records is not a filter problem; it is a cap the client never noticed.

The query

generated client paginated with page_size=100 but the API silently caps at 50  -  half the records never arrived

Steps

1. Measure the real page size

Request a page with your usual size and count the items returned. If you asked for 100 and got 50 with a next cursor present, the provider capped you. Check the response for a page-size or limit-echo field that states the applied value.

Expected: the actual items-per-page the provider honors, e.g. 50.

2. Find the documented maximum

Read the list endpoint's docs for the maximum page size. Providers usually document the cap somewhere - the generated code just never looked. Note whether the cap varies by endpoint or plan tier.

Expected: the documented max page size for the endpoint.

3. Stop assuming the requested size

Rewrite the client so pagination correctness never depends on page size: follow the next cursor until the end-of-list signal, whatever each page contains. Request the documented max (fewer requests), but treat the returned page as authoritative.

Expected: pagination code with no arithmetic that assumes a page holds N items.

4. Add a fetched-count sanity check

After a full pagination run, compare the collected item count against the provider's total (from a count field, a separate endpoint, or a known fixture). Log a warning when they disagree.

Expected: a count check that would have caught the missing half immediately.

5. Request the max, not a round number

Set the client's page size to the documented maximum rather than a habitual 100. Fewer pages means fewer requests, less rate-limit pressure, and faster syncs.

Expected: one config value per endpoint, set to the provider's documented max.

Use this when

• Full-list fetches return fewer items than expected with no errors
• The generated code sets a page size above the provider's documented max
• Returned pages are consistently smaller than requested with a next cursor present
• Halving or doubling the requested size changes nothing about the returned size

Not for this skill when

• The provider documents a fixed page size (then request it exactly)
• Items are missing because of filters or permissions (check the query)
• The cursor or loop logic is broken (fix termination first)
• The endpoint errors on oversized requests instead of capping (then lower the request)

Variant phrasings

half the records never arrived with no error

The signature of a silent cap: no error, no warning, just fewer items. Step 1 confirms it in one request.

API ignored the requested page size

Same fix. Some providers echo the applied size in the response - step 1 tells you to look for it.

Why it happens

The agent set page_size=100 as a sensible default and assumed the provider honors it, because most examples show the parameter being respected. The provider enforces a lower maximum server-side and, instead of erroring, silently returns fewer items with a valid next cursor. The client's loop terminates "successfully" having fetched a fraction of the list. Nothing failed, so nothing alerted - the data was just quietly incomplete.

Edge cases

• Cap differs by plan tier: the documented max may be for a higher tier than the account has. Step 1's measurement reflects the account's real cap.
• Provider errors instead of capping on some endpoints: handle both behaviors - read the error, lower the size, retry. Do not assume silent capping everywhere.
• Page size affects rate-limit cost: some providers count requests, not items. Larger pages are cheaper; step 5's max-size request is also the economical one.
• Cursor-based pagination makes size irrelevant: with a correct end-of-list signal, any page size fetches everything. Size only affects speed, not completeness.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_TUhlB852EhwnLTIofYEHCA