ERP export API paginated incorrectly mid-pull and the agent failed to detect the gap

TL;DR

Silent gaps are worse than failures because the agent reports success. Do not trust pagination to prove completeness: after the pull, compare per-day or per-page counts and totals against the ERP's own summary queries, and scan for continuity breaks in dates or sequence numbers. Where the export paginates by offset over changing data, switch to a stable keyset, such as created-at plus id, or pull in frozen slices so rows cannot drift between pages.

ERP export API paginated incorrectly mid-pull and the agent failed to detect the gap

Use this when

• Pulled totals are lower than the ERP's reported totals for the same period, with no error in the agent log.
• Certain dates or sequences are entirely absent from the pulled data.
• The export uses offset pagination and the underlying data changes during the pull.

Not for this skill when

• Totals disagree because of a filter difference, such as status or entity, between pull and summary. Align filters before blaming pagination.
• The ERP summary itself is wrong or cached. Verify the source with a second query type before rebuilding the pull.

Steps

1. Compute control totals from the ERP through an independent path: counts and sums per day or per account for the pulled range. Verify: control totals exist independently of the export being tested
2. Compare the pulled data slice by slice against the controls and localise the gap to specific slices or pages. Verify: the missing set is bounded to named slices
3. Inspect the pagination mechanics for those slices: offset versus cursor, sort stability, and whether records were inserted or updated mid-pull. Verify: a mechanism that explains the skip is identified
4. Refetch the missing slices with a stable method: keyset pagination or a frozen export, deduping on transaction id. Verify: refetched rows fill the gap with no duplicates at the edges
5. Add a standing completeness check to the job: no run reports success until slice totals tie to controls. Verify: a simulated gap now fails the run loudly instead of passing silently

Variant phrasings

ERP export pagination skipped records gap detection totals

The silent-gap phrasing.

agent pull incomplete offset pagination changing data

When switching to keyset pulls.

Compatibility: ERP and accounting export APIs with offset and cursor pagination, and agent data quality checks.

Why it happens

Offset pagination assumes a stable list: page two is meaningful only if the list did not change after page one. Live ledgers change constantly, and a row inserted or deleted early in the list shifts every later page, skipping or repeating a page-worth of rows at the seam. The API returns success for every page, so only an external completeness check can see the seam. Keyset pagination on an immutable sort key removes the assumption, which is why it is the standard cure.

Edge cases / pitfalls

• Updated timestamps that change when a row is merely touched make timestamp keysets drift too; pair the timestamp with the immutable id as a tiebreaker.
• A gap in the middle of a date range with clean edges is the offset-shift fingerprint; a gap at the end suggests an early termination instead.
• Control totals should come from a different endpoint or a report, not from summing the same export being validated.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_eNBe346Ez9G8X-8y9p7VxA