onboarding agent corrupted csv import partial write retried from row 0

Fix the onboarding agent corrupting CSV imports by retrying from row 0

TL;DR

A retry must resume from the last committed row, not from the top of the file. Checkpoint every batch you commit, dedupe on a stable external id, and make the whole import idempotent so re-running it is safe. Partial writes are normal; non-idempotent retries are the bug.

The problem

import run 1: wrote rows 0-412 of 2000, then timed out
import run 2 (retry): started at row 0 again
result: 413 users created twice, 1587 never created

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=onboarding agent corrupted csv import partial write retried from row 0"

Fix it

Step 1: Add a row cursor that survives the run

Write the last successfully committed row number to durable storage (a run record, a small table, a file - not agent memory) after every committed batch.

Expected: after a crash you can read the cursor and know exactly where the import stopped.

Step 2: Resume from the cursor, never from row 0

On retry, read the cursor first and skip every row at or below it. The first row processed must be cursor + 1.

Expected: re-running the import after a failure processes only the rows that never committed.

Step 3: Dedupe on a stable external id

Key every created user on an idempotent key derived from the CSV (employee id, email, or a row hash). Before creating, check whether that key already exists; if it does, skip or update instead of creating.

Expected: even a full re-run from row 0 produces zero duplicates.

Step 4: Commit in small batches with per-batch confirmation

Process in batches of 50 to 200 rows, and only advance the cursor after the target system confirms the batch. Never advance the cursor on a request you merely sent.

Expected: the cursor always reflects confirmed reality, so a crash between send and confirm replays at most one batch.

Step 5: Prove it with a kill mid-run

Start an import against a staging system, kill the agent halfway, and re-run.

Expected: zero duplicates, zero skipped rows, and the cursor matches the final committed row.

When this applies

• A failed import retry creates duplicate users
• The import has no recorded progress and always restarts at the top
• Large CSV imports (thousands of rows) time out or crash partway through

When it doesn't

• The CSV fails to parse at all (a parser problem, fix the parser first)
• Duplicates come from bad source data (two rows with the same email in the file)
• The target system itself duplicates on create (a target API problem)

Compatibility

Any HR or user CSV import into Okta, Entra ID, Google Workspace, or a custom SCIM endpoint. The pattern applies to every batch ETL the agent runs.

Variant phrasings

csv import retry duplicated users onboarding

Same fix. The retry path is the broken path; make it resume from the checkpoint.

onboarding agent re-imported the whole file after a partial failure

Check whether the agent even knows the import failed partway. Many agents see "error" and start over because nobody told them partial progress exists.

Why it happens

Agents treat an import as one atomic action because that is how it looks in the plan. In reality it is hundreds of writes, and the crash always lands in the middle. Restarting at row 0 feels like the safe choice, but without idempotent creates it writes every early row twice. The fix is bookkeeping the agent never had: a durable cursor and a dedupe key.

Edge cases

• Rows that were sent but never confirmed: reprocessing them is safe only if creates are keyed
• The CSV changed between runs: hash the file and refuse to resume if it changed
• Concurrent runs of the same import: lock the run id so two agents cannot import at once

If it still fails

• Capture the cursor value, the row range of each batch, and the dedupe key scheme before changing anything else.
• Reproduce with a small 200-row file so you are not debugging a crowd.
• Check the target system status page; API outages look exactly like import bugs.

Prevention

• Every batch job the agent runs gets a durable cursor from day one.
• Every create the agent performs gets an idempotency key from day one.
• Test the kill-and-resume path for every importer before it touches production.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstBxPiN1AooEjtFSfCIVyfw