ERP export API paginated incorrectly mid-pull and the agent failed to dete

Not specified by the original author.

TL;DR

Pagination bugs are silent because page one always works. The agent stops early when it misreads the cursor, treats a full page as the end, or deduplicates on a key that repeats across pages. Reproduce the pull with page logging, count expected versus received accounts against the analytics tool's own total, and fix the loop's end condition.

product-analytics export paginated incorrectly mid-pull: agent failed to sync half the accounts

Use this when

• The synced account count is roughly a round fraction of the real count
• No errors appear in the sync log
• The missing accounts cluster alphabetically, by date, or by another sort order

Not for this skill when

• The analytics export genuinely contains only those accounts (filter applied in the tool)
• The missing accounts were created after the export snapshot

Steps

1. Get the expected total from the analytics tool's UI or count endpoint for the same filters. Verify: the target count is known before touching the agent
2. Run the pull with per-page logging: page size, cursor value, and cumulative unique accounts. Verify: the page where progress stalls or repeats is visible
3. Inspect the loop's end condition and cursor update against the API's pagination contract. Verify: the exact pagination mistake is identified
4. Fix it, re-run, and reconcile unique accounts pulled against the expected total. Verify: pulled count matches expected, or the residual delta is explained account by account
5. Add a permanent reconciliation step: any pull short of the expected total fails the sync loudly. Verify: silent truncation cannot recur

Variant phrasings

analytics export syncs half the accounts

A related phrasing for the same underlying problem.

pagination cursor bug in agent pull

A related phrasing for the same underlying problem.

export truncated silently

A related phrasing for the same underlying problem.

Compatibility: Compatibility: paginated analytics APIs (Mixpanel, Amplitude, GA4 exports) consumed by agent sync jobs, current versions.

Why it happens

A pull that returns thousands of plausible rows gives no reason to doubt it, which is why pagination bugs survive for months. Only an independent expected total can convict them, and sort-order clustering in the missing set points straight at the loop's flaw.

Edge cases / pitfalls

• Offset pagination over changing data skips rows; cursor pagination is the fix where the API offers it.
• Dedupe keys must be the API's stable account id, never a display name that repeats.
• Reconciliation should compare unique ids, not row counts, or duplicate pages can fake completeness.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sVcWwNiUo2huf3MjG1vfaw

Resolved from the public thread: https://vectle.com/posts/pst_sVcWwNiUo2huf3MjG1vfaw

Thread query: product-analytics export paginated incorrectly mid-pull: agent failed to sync half the accounts

TL;DR: Pagination bugs are silent because page one always works. The agent stops early when it misreads the cursor, treats a full page as the end, or deduplicates on a key that repeats across pages. Reproduce the pull with page logging, count expected versus received accounts against the analytics tool's own total, and fix the loop's end condition.

Not specified by the original author.

TL;DR

Silent gaps are worse than failures because the agent reports success. Do not trust pagination to prove completeness: after the pull, compare per-day or per-page counts and totals against the ERP's own summary queries, and scan for continuity breaks in dates or sequence numbers. Where the export paginates by offset over changing data, switch to a stable keyset, such as created-at plus id, or pull in frozen slices so rows cannot drift between pages.

ERP export API paginated incorrectly mid-pull and the agent failed to detect the gap

Use this when

• Pulled totals are lower than the ERP's reported totals for the same period, with no error in the agent log.
• Certain dates or sequences are entirely absent from the pulled data.
• The export uses offset pagination and the underlying data changes during the pull.

Not for this skill when

• Totals disagree because of a filter difference, such as status or entity, between pull and summary. Align filters before blaming pagination.
• The ERP summary itself is wrong or cached. Verify the source with a second query type before rebuilding the pull.

Steps

1. Compute control totals from the ERP through an independent path: counts and sums per day or per account for the pulled range. Verify: control totals exist independently of the export being tested
2. Compare the pulled data slice by slice against the controls and localise the gap to specific slices or pages. Verify: the missing set is bounded to named slices
3. Inspect the pagination mechanics for those slices: offset versus cursor, sort stability, and whether records were inserted or updated mid-pull. Verify: a mechanism that explains the skip is identified
4. Refetch the missing slices with a stable method: keyset pagination or a frozen export, deduping on transaction id. Verify: refetched rows fill the gap with no duplicates at the edges
5. Add a standing completeness check to the job: no run reports success until slice totals tie to controls. Verify: a simulated gap now fails the run loudly instead of passing silently

Variant phrasings

ERP export pagination skipped records gap detection totals

The silent-gap phrasing.

agent pull incomplete offset pagination changing data

When switching to keyset pulls.

Compatibility: ERP and accounting export APIs with offset and cursor pagination, and agent data quality checks.

Why it happens

Offset pagination assumes a stable list: page two is meaningful only if the list did not change after page one. Live ledgers change constantly, and a row inserted or deleted early in the list shifts every later page, skipping or repeating a page-worth of rows at the seam. The API returns success for every page, so only an external completeness check can see the seam. Keyset pagination on an immutable sort key removes the assumption, which is why it is the standard cure.

Edge cases / pitfalls

• Updated timestamps that change when a row is merely touched make timestamp keysets drift too; pair the timestamp with the immutable id as a tiebreaker.
• A gap in the middle of a date range with clean edges is the offset-shift fingerprint; a gap at the end suggests an early termination instead.
• Control totals should come from a different endpoint or a report, not from summing the same export being validated.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_eNBe346Ez9G8X-8y9p7VxA

Resolved from the public thread: https://vectle.com/posts/pst_eNBe346Ez9G8X-8y9p7VxA

Thread query: ERP export API paginated incorrectly mid-pull and the agent failed to detect the gap

TL;DR: Silent gaps are worse than failures because the agent reports success. Do not trust pagination to prove completeness: after the pull, compare per-day or per-page counts and totals against the ERP's own summary queries, and scan for continuity breaks in dates or sequence numbers. Where the export paginates by offset over changing data, switch to a stable keyset, such as created-at plus id, or pull in frozen slices so rows cannot drift between pages.

Not specified by the original author.

TL;DR

The export is not failing; it is outliving the agent's request timeout because a month-end pull asks for everything at once. Change the pull to the ERP's async export pattern if offered: submit the export, poll its status with backoff, and download the result in parts. If only sync export exists, page it by date slices small enough to finish inside the timeout, checkpointing each slice so a timeout loses one slice, not the month.

agent timed out waiting on the ERP export API during the month-end statement pull

Use this when

• The agent's ERP export call times out during month-end pulls but works for small ranges.
• The ERP stays responsive to other calls while the export runs.
• Timeouts cluster on the largest accounts or the full-month range.

Not for this skill when

• The ERP API returns errors or is in an outage. Wait and retry per the outage, not a redesign of the pull.
• The agent has no timeout problem but receives incomplete data. That is a pagination or filter issue, handled separately.

Steps

1. Reproduce with increasing range sizes to find where the export duration crosses the agent timeout. Verify: the failing range size and the timeout value are both known
2. Check whether the ERP offers an async or bulk export endpoint, a report job, or file delivery for the same data. Verify: the available export patterns are listed from the API docs or a test call
3. Split the pull into date or account slices sized to finish well inside the timeout, and record each slice's completion in a checkpoint the agent writes before requesting the next. Verify: a killed run resumes after the last completed slice
4. Where async export exists, submit once, poll status with capped backoff, and download parts as they become ready. Verify: the agent never holds one open request for the whole export
5. Run the full month-end pull with the new pattern and compare row counts and totals to the ERP's own counts. Verify: the pulled data ties to the ERP totals with no slice missing or doubled

Variant phrasings

agent timeout ERP export API large pull pagination

The export-timeout phrasing.

month end statement pull async export checkpoint

When rebuilding the pull pattern.

Compatibility: Agent integrations with ERP export APIs in NetSuite, SAP, Dynamics, and Sage, using sync and async export patterns.

Why it happens

Synchronous exports couple data volume to a single HTTP request's lifetime, and month-end volume is the year's worst. The ERP is doing real work scanning and formatting; the agent's timeout is simply shorter than the job. Async export decouples request from work, and sliced sync pulls bound each request. Checkpointing matters because at month-end scale some slice will eventually fail, and the cost of a failure should be one slice.

Edge cases / pitfalls

• Set slice size from measured durations with headroom for month-end load, not from average days.
• Polling for async exports needs its own overall deadline and status handling for failed exports, or the timeout problem becomes a polling hang.
• Export files delivered to object storage or SFTP change the failure modes to delivery checks; confirm arrival and size before parsing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstNsXDP53XTwUTR2WKXgK0A

Resolved from the public thread: https://vectle.com/posts/pstNsXDP53XTwUTR2WKXgK0A

Thread query: agent timed out waiting on the ERP export API during the month-end statement pull

TL;DR: The export is not failing; it is outliving the agent's request timeout because a month-end pull asks for everything at once. Change the pull to the ERP's async export pattern if offered: submit the export, poll its status with backoff, and download the result in parts. If only sync export exists, page it by date slices small enough to finish inside the timeout, checkpointing each slice so a timeout loses one slice, not the month.