agent's access-review export truncated: identity API pagination

agent's access-review export truncated: identity API pagination

TL;DR

Identity APIs page their results, often 50 or 100 per page, and the agent read page one and called it complete. Follow the pagination cursor (next page token or Link header) until the API says there are no more pages, then compare the collected count against the API's reported total before starting the review. A review run on a partial export certifies access nobody looked at.

agent's access-review export truncated: identity API pagination

Use this when

• An access review covers fewer users than the IdP dashboard shows
• The export has a suspiciously round count (exactly 100, 200, 1000)
• The agent's user list ends mid-alphabet
• You are pulling any large list from an identity API (users, groups, app assignments)

Not for this skill when

• You are using the IdP's built-in report builder (it handles paging for you)
• You are designing the access-review policy itself
• You are setting up SCIM provisioning

Steps

1. Check the API response for the pagination mechanism: a nextPageToken or cursor field, or a Link header with rel next. Expected: you find the paging field instead of assuming a single page.

2. Loop: request the next page until the API returns no cursor.

   records = []
   cursor = None
   while True:
       page = fetch_page(cursor)  # your API call here
       records.extend(page["items"])
       cursor = page.get("next_cursor")
       if not cursor:
           break
   print("collected", len(records), "expected", page.get("total"))

   Expected: the loop ends on an empty cursor, not on a guess.

3. Reconcile: compare the collected count with the total the API reports. Expected: the counts match; if they do not, something was skipped.

4. Only then run the review against the full set. Expected: the review covers every identity, and the count is on the record.

Variant: Okta API only returned first 200 users

Okta defaults to 200 per page with a Link header for the next page. Follow the Link headers to the end; the System Log API pages the same way.

Variant: access review export missing users

Before re-pulling, check whether the missing users were filtered (deprovisioned, suspended) rather than paged out. Then fix the paging and compare counts.

Variant: identity API pagination cursor loop

Some APIs use opaque cursors that expire; if a cursor stops working mid-pull, restart the pull rather than stitching partial results, and keep the page size stable.

Why this happens

APIs cap page size to protect themselves, and many SDKs return one page per call without complaint. An agent that does not know to look for the cursor gets a silent subset, and the round number (exactly 100) is the tell that nobody noticed.

Edge cases and pitfalls

• Data can change mid-pagination (users added or removed); use a stable sort or a snapshot parameter if the API offers one.
• Many pages means many requests; respect rate limits with backoff instead of hammering the API.
• Some APIs cap the total retrievable pages; for very large directories, pull in segments (by group or by letter range).
• Not every API returns a total; when it does not, pull twice and compare, or cross-check against the IdP dashboard count.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GSk9sYR-vJKGX4Cb2y0Nyw