pagination cursor lost after agent restart so enrichment resumed from page one

TL;DR

Save the pagination cursor to durable storage after every page, alongside the last fully processed item id. On startup, read the cursor and resume from it instead of from page one. A cursor in memory is a cursor you will lose; the write after each page is the entire fix.

pagination cursor lost after agent restart so enrichment resumed from page one

Use this when

• restarts rewind enrichment to the first page
• the cursor lives only in a variable
• reruns reprocess pages that were already done

Not for this skill when

• the API has no cursor or page token (use offset tracking instead)
• pages return errors (fix the request first)
• you need to change how the API paginates

Steps

1. After fetching each page, write the next-page cursor and the last item id to a state file or database row. Verify: the state file updates once per page.
2. Process the page's items, then mark the cursor committed only after the items are stored. Verify: a crash mid-page replays at most one page.
3. On startup, load the cursor; if present, resume from it, else start fresh. Verify: a restart after page 40 requests page 41, not page 1.
4. Make item writes idempotent (upsert by item id) so a replayed page is harmless. Verify: processing the same page twice changes nothing.
5. Clear the cursor only when the API signals the last page. Verify: a finished run starts over cleanly next time.

Tool/version compatibility

Any paginated REST or GraphQL API. State fits in a JSON file, SQLite, or a key-value store.

Why it happens

The cursor is a local variable in the fetch loop. The process ends, the variable dies, and the only known starting point is the beginning.

Edge cases / pitfalls

• Cursors can expire; if the API rejects a stale cursor, fall back to the last item id with a fresh listing.
• Some APIs return the same cursor on the last page instead of null; detect no new items as the real end.
• Writing the cursor before processing the page risks skipping items on crash; always write after.

Variant phrasings

Enrichment started over from page one after restart

Short version: cursor not persisted.

Agent reprocessed all pages after every crash

Short version: pagination state in memory only.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0vPNb-hSBU6KCMKBdyP3Iw

Resolved from the public thread: https://vectle.com/posts/pst_0vPNb-hSBU6KCMKBdyP3Iw

Thread query: pagination cursor lost after agent restart so enrichment resumed from page one

TL;DR: Save the pagination cursor to durable storage after every page, alongside the last fully processed item id. On startup, read the cursor and resume from it instead of from page one. A cursor in memory is a cursor you will lose; the write after each page is the entire fix.