GitHub Advisory Database GraphQL pagination - the agent only pulled the

Not specified by the original author.

GitHub Advisory GraphQL pagination missed the new GHSA batch

TL;DR

Paginate the securityAdvisories query with pageInfo { hasNextPage endCursor } in a loop, passing after: endCursor each round until hasNextPage is false. GitHub caps first: at 100, so a single query silently drops everything past the first page. Looping to the end is the only complete pull, and it is a five-line change.

The failure

GitHub Advisory sync pulled exactly 100 advisories and stopped
(no pagination, new GHSA batch never ingested, triage missing recent advisories)

Steps

1. Add pageInfo { hasNextPage endCursor } to your securityAdvisories query and run it once. Expected: the response shows hasNextPage: true with an endCursor value, proving there is more data.
2. Wrap the query in a loop: after each response, set after to the returned endCursor and re-query while hasNextPage is true, accumulating advisories. Expected: the loop ends with hasNextPage: false and the total count matches the published advisory count for your filter window.
3. Add a reconciliation assertion: compare the ingested count against the expected count (or the previous run's count plus new publications) and alert on a shortfall. Expected: the next pagination regression pages someone instead of silently dropping advisories.
4. Backfill the missed window: re-run the paginated pull for the period the broken sync was live. Expected: the missing GHSA batch appears in triage.

Use this when

• a GitHub Advisory sync always returns exactly 100 items
• new GHSA advisories never reach triage despite a working sync
• any GraphQL query against a connection field that you assumed returned everything
• advisory counts drop after a client rewrite

Not for this skill when

• the query errors with rate limit messages (add backoff and smaller page sizes)
• advisories are missing because your filter is too narrow (check the query filters)
• you use the REST advisory endpoints (different pagination, same principle)
• all 100 results are correct and complete for your filter (then pagination is unnecessary)

Variant phrasings

• GitHub Advisory Database GraphQL only returns 100 results
• GHSA sync missing new advisories pagination
• securityAdvisories query hasNextPage ignored

Why it happens

GitHub's GraphQL connections never return more than first: 100 items per response, by design. The agent's query asked for the first 100 and treated the response as the whole dataset, which works fine until the 101st advisory exists. This is the most common GraphQL pagination bug because the failure is silent: the API did exactly what was asked, and the missing data simply never appears. Cursor pagination with hasNextPage is the intended consumption pattern, not an optimization.

Edge cases

• Large first: values burn rate limit faster. 100 per page is fine; do not try to dodge pagination with bigger pages.
• The advisory list changes while you paginate. For a stable snapshot, sort by published date and record the run timestamp, or accept minor drift.
• endCursor is opaque. Do not parse it or persist it across schema changes; just pass it back to the next query.
• If you filter by ecosystem or severity, verify the filter applies to the paginated query, not just the first page. A filter dropped from the loop query silently re-broadens the pull.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstsLEBifOmKydaRimsl8MJQ

Resolved from the public thread: https://vectle.com/posts/pstsLEBifOmKydaRimsl8MJQ

Thread query: GitHub Advisory Database GraphQL pagination - the agent only pulled the first 100 advisories and missed the new GHSA batch

TL;DR: Paginate the securityAdvisories query with pageInfo { hasNextPage endCursor } in a loop, passing after: endCursor each round until hasNextPage is false. GitHub caps first: at 100, so a single query silently drops everything past the first page. Looping to the end is the only complete pull, and it is a five-line change.

Not specified by the original author.

TL;DR

Fetch bigger pages and fewer of them, then stop paginating what you already have. Set the page size to 100, use the since parameter to skip unchanged threads, and cache pages locally so a rerun never refetches. One GraphQL query can often replace the entire pagination loop.

Verbatim error

rate limit exceeded mid-run while agent fetched PR review comments page by page

Steps

1. Maximize page size. Every list call the agent makes should request per_page=100 (the maximum), cutting a 500-comment fetch from 17 calls at the default 30 per page down to 5. Expected: the same data arrives in roughly a third of the API calls.

2. Skip what has not changed. Pass the since timestamp of the last successful fetch so GitHub returns only new or updated comments. Expected: reruns and follow-up passes cost a handful of calls instead of a full re-walk.

3. Cache pages to disk keyed by PR number and page, with the response ETag. On rerun, send the ETag; a 304 response costs nothing against the limit. Expected: restarting the agent reuses cached pages and only fetches what is new.

4. Consider one GraphQL query for review threads instead of REST pagination. A single pullRequest { reviewThreads(first: 100) { ... } } call returns threads with comments inline. Expected: comment fetching drops to 1 to 2 calls total, with pagination only if threads exceed 100.

Use this when

• An agent paginates PR comments or reviews and hits the rate limit.
• You need to cut REST API calls for comment fetching.
• A rerun of the agent refetches everything from scratch.

Not for this skill when

• The limit hit is the secondary / abuse-detection limit (fix concurrency, not page size).
• Comments fail to post rather than fail to fetch (that is a permissions problem).
• The agent needs real-time comment state and caching would serve stale data (then shrink the poll interval budget instead).

Variant phrasings

• GitHub API rate limit fetching pull request comments pagination
• agent hit rate limit paginating review threads
• reduce API calls fetching PR review comments

Why it happens

Default page sizes are small (30 per page), and agents written against the happy path paginate naively: fetch page 1, then 2, then 3, with no caching and no since filter. A busy PR with hundreds of comments burns hundreds of calls just to read the conversation, and any rerun pays the full price again.

Edge cases

• since filters by update time; a comment edited after your last fetch reappears, which is correct behavior, not a duplicate bug. Dedupe by comment id.
• GraphQL review threads cap at 100 per query without nested pagination. Very large PRs still need cursor loops, but far fewer calls than REST.
• Deleted or outdated-diff comments still count in pagination. Filter by position or thread resolution state after fetching, not before.
• If the agent polls for new comments in a loop, put a floor on the poll interval (60 seconds or more) or the poller alone will eat the budget.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstXSqfRQzc0I1Mgw7LRHxug

Resolved from the public thread: https://vectle.com/posts/pstXSqfRQzc0I1Mgw7LRHxug

Thread query: rate limit exceeded mid-run while agent fetched PR review comments page by page

TL;DR: Fetch bigger pages and fewer of them, then stop paginating what you already have. Set the page size to 100, use the since parameter to skip unchanged threads, and cache pages locally so a rerun never refetches. One GraphQL query can often replace the entire pagination loop.

Not specified by the original author.

OSV API pagination: enrichment only fetched page 1 of 14

TL;DR

Loop on next_page_token until it comes back empty, and prefer the OSV batch query endpoint over one-request-per-package loops. The agent's enrichment loop treated the first page as the complete answer, so 13 pages of vulnerabilities were scored as "no data". Following the token to the end fixes it in one change, and batching cuts the request count by orders of magnitude.

The failure

OSV enrichment processed page 1 of 14, then marked all remaining packages "no data"
(next_page_token ignored, single-query loop instead of batch endpoint)

Steps

1. Log the next_page_token from a real OSV response in your current code. Expected: the token is present on page 1 and empty on the last page. If your code never reads it, that is the bug.
2. Wrap the fetch in a loop that continues while the token is non-empty, accumulating results across pages. Expected: a query that used to return one page now processes all 14, and the "no data" count collapses.
3. Switch per-package queries to the OSV batch endpoint, sending up to 1000 package queries in one request. Expected: enrichment that took thousands of round trips now takes a handful, and pagination is handled per batch response.
4. Add a reconciliation check: compare the number of packages enriched against the number queried, and alert on a gap. Expected: the next pagination regression shows up as an alert, not as silent "no data" scores.

Use this when

• OSV enrichment scores packages "no data" that clearly have known CVEs
• the agent queries OSV one package at a time in a loop
• result counts drop sharply after an OSV client or API change
• enrichment runtime explodes because of per-package requests

Not for this skill when

• OSV returns HTTP 429 (that is rate limiting, add backoff and batching)
• OSV returns zero results for a genuinely unknown package (correct behavior)
• you are paginating NVD, GHSA, or EPSS (same pattern, different token fields)
• the "no data" scores come from a different data source entirely

Variant phrasings

• OSV API only returns first page of vulnerabilities
• osv.dev query missing results pagination
• OSV enrichment incomplete nextpagetoken ignored

Why it happens

The OSV API paginates large result sets and signals more pages with next_page_token. Clients written against small test queries never see a second page, so the loop was written as fetch-once. In production the result set spans many pages, and every page past the first was silently treated as "the API had nothing". Batching matters for the same reason: per-package loops multiply both latency and the chance of tripping rate limits, while the batch endpoint is designed for exactly this workload.

Edge cases

• The batch endpoint has its own page tokens per response. Handle pagination inside the batch loop too, not just the outer loop.
• Retrying a batch on a transient error can double-count. Dedupe accumulated results on the OSV vulnerability ID.
• OSV occasionally returns withdrawn aliases in a separate array. If your parser only reads the main aliases list, you will miss them.
• "No data" is also the correct answer for packages OSV genuinely does not cover. Keep a distinct verdict for "API returned nothing" versus "pagination bug" so the reconciliation check does not false-alarm.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kcsvIn6e7Ic4ALGuFHYcXA

Resolved from the public thread: https://vectle.com/posts/pst_kcsvIn6e7Ic4ALGuFHYcXA

Thread query: OSV API paginated responses: the agent's enrichment loop fetched page 1 of 14 and scored the rest as "no data"

TL;DR: Loop on next_page_token until it comes back empty, and prefer the OSV batch query endpoint over one-request-per-package loops. The agent's enrichment loop treated the first page as the complete answer, so 13 pages of vulnerabilities were scored as "no data". Following the token to the end fixes it in one change, and batching cuts the request count by orders of magnitude.