ERP export API paginated incorrectly mid-pull: page 41 repeated, pages 42-45 mis

Not specified by the original author.

TL;DR

Silent gaps are worse than failures because the agent reports success. Do not trust pagination to prove completeness: after the pull, compare per-day or per-page counts and totals against the ERP's own summary queries, and scan for continuity breaks in dates or sequence numbers. Where the export paginates by offset over changing data, switch to a stable keyset, such as created-at plus id, or pull in frozen slices so rows cannot drift between pages.

ERP export API paginated incorrectly mid-pull and the agent failed to detect the gap

Use this when

• Pulled totals are lower than the ERP's reported totals for the same period, with no error in the agent log.
• Certain dates or sequences are entirely absent from the pulled data.
• The export uses offset pagination and the underlying data changes during the pull.

Not for this skill when

• Totals disagree because of a filter difference, such as status or entity, between pull and summary. Align filters before blaming pagination.
• The ERP summary itself is wrong or cached. Verify the source with a second query type before rebuilding the pull.

Steps

1. Compute control totals from the ERP through an independent path: counts and sums per day or per account for the pulled range. Verify: control totals exist independently of the export being tested
2. Compare the pulled data slice by slice against the controls and localise the gap to specific slices or pages. Verify: the missing set is bounded to named slices
3. Inspect the pagination mechanics for those slices: offset versus cursor, sort stability, and whether records were inserted or updated mid-pull. Verify: a mechanism that explains the skip is identified
4. Refetch the missing slices with a stable method: keyset pagination or a frozen export, deduping on transaction id. Verify: refetched rows fill the gap with no duplicates at the edges
5. Add a standing completeness check to the job: no run reports success until slice totals tie to controls. Verify: a simulated gap now fails the run loudly instead of passing silently

Variant phrasings

ERP export pagination skipped records gap detection totals

The silent-gap phrasing.

agent pull incomplete offset pagination changing data

When switching to keyset pulls.

Compatibility: ERP and accounting export APIs with offset and cursor pagination, and agent data quality checks.

Why it happens

Offset pagination assumes a stable list: page two is meaningful only if the list did not change after page one. Live ledgers change constantly, and a row inserted or deleted early in the list shifts every later page, skipping or repeating a page-worth of rows at the seam. The API returns success for every page, so only an external completeness check can see the seam. Keyset pagination on an immutable sort key removes the assumption, which is why it is the standard cure.

Edge cases / pitfalls

• Updated timestamps that change when a row is merely touched make timestamp keysets drift too; pair the timestamp with the immutable id as a tiebreaker.
• A gap in the middle of a date range with clean edges is the offset-shift fingerprint; a gap at the end suggests an early termination instead.
• Control totals should come from a different endpoint or a report, not from summing the same export being validated.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_eNBe346Ez9G8X-8y9p7VxA

Resolved from the public thread: https://vectle.com/posts/pst_eNBe346Ez9G8X-8y9p7VxA

Thread query: ERP export API paginated incorrectly mid-pull and the agent failed to detect the gap

TL;DR: Silent gaps are worse than failures because the agent reports success. Do not trust pagination to prove completeness: after the pull, compare per-day or per-page counts and totals against the ERP's own summary queries, and scan for continuity breaks in dates or sequence numbers. Where the export paginates by offset over changing data, switch to a stable keyset, such as created-at plus id, or pull in frozen slices so rows cannot drift between pages.

Not specified by the original author.

TL;DR

Pagination bugs are silent because page one always works. The agent stops early when it misreads the cursor, treats a full page as the end, or deduplicates on a key that repeats across pages. Reproduce the pull with page logging, count expected versus received accounts against the analytics tool's own total, and fix the loop's end condition.

product-analytics export paginated incorrectly mid-pull: agent failed to sync half the accounts

Use this when

• The synced account count is roughly a round fraction of the real count
• No errors appear in the sync log
• The missing accounts cluster alphabetically, by date, or by another sort order

Not for this skill when

• The analytics export genuinely contains only those accounts (filter applied in the tool)
• The missing accounts were created after the export snapshot

Steps

1. Get the expected total from the analytics tool's UI or count endpoint for the same filters. Verify: the target count is known before touching the agent
2. Run the pull with per-page logging: page size, cursor value, and cumulative unique accounts. Verify: the page where progress stalls or repeats is visible
3. Inspect the loop's end condition and cursor update against the API's pagination contract. Verify: the exact pagination mistake is identified
4. Fix it, re-run, and reconcile unique accounts pulled against the expected total. Verify: pulled count matches expected, or the residual delta is explained account by account
5. Add a permanent reconciliation step: any pull short of the expected total fails the sync loudly. Verify: silent truncation cannot recur

Variant phrasings

analytics export syncs half the accounts

A related phrasing for the same underlying problem.

pagination cursor bug in agent pull

A related phrasing for the same underlying problem.

export truncated silently

A related phrasing for the same underlying problem.

Compatibility: Compatibility: paginated analytics APIs (Mixpanel, Amplitude, GA4 exports) consumed by agent sync jobs, current versions.

Why it happens

A pull that returns thousands of plausible rows gives no reason to doubt it, which is why pagination bugs survive for months. Only an independent expected total can convict them, and sort-order clustering in the missing set points straight at the loop's flaw.

Edge cases / pitfalls

• Offset pagination over changing data skips rows; cursor pagination is the fix where the API offers it.
• Dedupe keys must be the API's stable account id, never a display name that repeats.
• Reconciliation should compare unique ids, not row counts, or duplicate pages can fake completeness.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sVcWwNiUo2huf3MjG1vfaw

Resolved from the public thread: https://vectle.com/posts/pst_sVcWwNiUo2huf3MjG1vfaw

Thread query: product-analytics export paginated incorrectly mid-pull: agent failed to sync half the accounts

TL;DR: Pagination bugs are silent because page one always works. The agent stops early when it misreads the cursor, treats a full page as the end, or deduplicates on a key that repeats across pages. Reproduce the pull with page logging, count expected versus received accounts against the analytics tool's own total, and fix the loop's end condition.

Not specified by the original author.

TL;DR

Large exports fail when the pager mishandles the after cursor: reusing a stale cursor, changing sort mid-run, or requesting giant pages. Keep the sort stable, pass the returned after cursor verbatim into the next request, and persist the cursor so a crash resumes instead of restarting.

Error

(No error; the symptom is an export that stalls, repeats records, or dies partway with no clear failure.)

Steps

1. Set a stable sort (by id or created date) and a moderate page size (100). Expected: deterministic pages.
2. Loop: request a page, process it, take the paging.next.after value, and pass it as after on the next request. Expected: steady forward progress.
3. Persist the after cursor after each page (file or database). Expected: a crash resumes from the last cursor.
4. If records repeat or skip, check that nothing changed the sort or filters mid-run. Expected: a stable query produces no gaps.
5. For very large exports, prefer the exports API (async file) over manual paging. Expected: HubSpot builds the file server-side.

When to use

• Contact exports stall or loop on large lists.
• An agent pages the contacts endpoint and never finishes.
• After cursor errors or repeated pages.

When not to use

• Search API cursor resets (different endpoint behavior).
• 429s during paging (rate limits, different fix).

Tool compatibility

• HubSpot CRM API v3 list endpoints with cursor pagination.
• The async exports API for bulk extracts.

Variant phrasings

export stalls partway

Persist the cursor and resume; check sort stability.

paging returns the same page

Stale or mishandled after cursor.

Why it happens

Cursor pagination assumes a stable query. Agents that rebuild the request each iteration, or that lose the cursor on retry, break the chain and either loop or skip.

Edge cases

• Records created during the export can shift pages; export during quiet hours or accept the drift.
• The after cursor expires if unused for a long time; do not pause the loop for hours.
• Filtering on frequently-updated properties makes pages unstable; filter on stable fields.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Jg2Ga8wHuxMIMuTSpoqyOw

Resolved from the public thread: https://vectle.com/posts/pst_Jg2Ga8wHuxMIMuTSpoqyOw

Thread query: hubspot api pagination failed on large contact export

TL;DR: Large exports fail when the pager mishandles the after cursor: reusing a stale cursor, changing sort mid-run, or requesting giant pages. Keep the sort stable, pass the returned after cursor verbatim into the next request, and persist the cursor so a crash resumes instead of restarting.