onboarding agent timed out 10k row hr csv left half-provisioned users

Fix the onboarding agent timing out on a 10k-row HR CSV and leaving half-provisioned users

TL;DR

Split the import into small committed batches with a durable checkpoint after each one, and run it as a background job that can resume. A 10k-row import is not one action; it is hundreds of small ones. Timeouts are inevitable at that scale, so the import must survive them instead of fearing them.

The problem

import: 10,000 rows, single synchronous run
result: timed out at row 4,812 after 30 minutes
state: 4,812 users created, 5,188 missing, no record of where it stopped
retry: starts at row 0, duplicates the first 4,812

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=onboarding agent timed out 10k row hr csv left half-provisioned users"

Fix it

Step 1: Chunk the file into small batches

Split the CSV into batches of 100 to 200 rows. Each batch is an independent unit of work with its own commit and its own confirmation.

Expected: no single unit of work runs long enough to hit a timeout.

Step 2: Checkpoint after every committed batch

Write the last committed row number to durable storage after each batch confirms. The checkpoint must survive the agent crashing, the VM restarting, and the run being killed.

Expected: a crash at any point leaves an exact resume position.

Step 3: Run the import as a resumable background job

Move the import out of the interactive agent turn into a background job that processes batches and updates the checkpoint. The agent monitors progress; it does not hold the whole import in one turn.

Expected: timeouts of the agent turn no longer affect the import at all.

Step 4: Make every create idempotent

Key each user create on a stable id from the CSV (employee id or email) and check existence before creating. A resumed batch that replays a row must not duplicate it.

Expected: re-running any batch is safe, including the batch that was in flight during the crash.

Step 5: Prove resume works by killing it mid-run

Start the import in staging, kill the job at 40 percent, and resume.

Expected: the import completes with zero duplicates, zero skipped rows, and the final checkpoint matches the row count.

When this applies

• Large CSV imports time out partway and leave partial user sets
• The retry restarts from the top and duplicates early rows
• The import runs inside a single agent turn with a time limit

When it doesn't

• The CSV fails to parse (fix the parser first)
• The source data has duplicate rows (dedupe the file first)
• A small import fails on every row (a per-row bug, not a scale problem)

Compatibility

HR CSV imports into Okta, Entra ID, Google Workspace, and BambooHR-style sources; any batch user creation the agent performs at scale.

Variant phrasings

hr import timed out halfway thousands of users

Same fix. The timeout is a scale fact; the missing checkpoint is the bug.

bulk user import left half the users unprovisioned

Reconcile first: diff the CSV against the created users to find the exact gap, then resume from the checkpoint.

Why it happens

Agents process files the way they read them: start at the top, work down, finish. That works for 50 rows. At 10k rows the run outlives every timeout in the stack - the agent turn, the HTTP request, the function limit - and it always dies in the middle. Without a checkpoint the only recovery is starting over, and without idempotent creates starting over means duplicates. The import was designed for the happy path at a scale where the happy path never happens.

Edge cases

• The CSV changes between runs: hash the file and refuse to resume on a different hash
• Rows near the crash may have been sent but not confirmed: idempotent creates make replay safe
• Rate limits on the target API: pace batches so the import does not 429 itself to death

If it still fails

• Capture the checkpoint value, the batch size, and the row range of the failed batch before changing anything else.
• Reproduce with a 500-row file so you are not debugging 10k rows.
• Check the target system status page; API outages look exactly like import bugs.

Prevention

• Any import over 500 rows gets chunking, checkpoints, and idempotent creates from day one.
• Progress dashboards show checkpoint vs total so stalls are visible.
• Alert when a batch job makes no checkpoint progress for 15 minutes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-f6gjAXZPAtWe9VyA1IkWQ