enrichment pipeline timed out after 200 of 5000 rows with no checkpoint

TL;DR

Write each processed row to the output store (or a checkpoint file) as soon as it finishes, plus a running last-row marker. On restart, skip every row at or below the marker. A timeout then costs you one row, not the whole run. The marker is the whole trick; everything else is plumbing.

enrichment pipeline timed out after 200 of 5000 rows with no checkpoint

Use this when

• a pipeline dies partway and restarts from row one
• you re-enrich rows that were already done
• timeouts or restarts wipe in-memory progress

Not for this skill when

• the enrichment API itself is down (fix the call first)
• you need to pick an enrichment vendor
• rows fail because the input data is bad (clean the input)

Steps

1. Add a processed_at column or a checkpoint file holding the last finished row id. Verify: after 5 rows the marker shows row 5.
2. Process rows in small batches (25 to 50), committing the checkpoint after each batch. Verify: kill the process mid-run and the marker is within one batch of reality.
3. On startup, query for rows without processed_at and begin there. Verify: a second run touches zero already-done rows.
4. Make each row operation idempotent: upsert by row id, never blind insert. Verify: running the same batch twice changes nothing.
5. Set a per-row timeout with a retry budget of 3 on 5xx and timeouts only, then mark the row failed and move on. Verify: one bad row cannot stall the run.

Tool/version compatibility

Any stack: Python scripts, Airflow, or plain cron. Works with Postgres, SQLite, or a flat JSONL checkpoint file.

Why it happens

The pipeline holds progress in memory and writes results only at the end. Any interruption discards everything, so the default recovery is a full restart.

Edge cases / pitfalls

• A checkpoint write that fails silently is worse than none: verify the write landed.
• Clock skew can confuse timestamp markers; prefer monotonic row ids.
• Marking rows failed still counts as progress: a poison row should not be retried forever.

Variant phrasings

Pipeline kept reprocessing the same rows after every timeout

Short version: no checkpoint between runs.

Long enrichment job never finishes because it restarts

Short version: progress not persisted per row.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_N0deI-7N-x1oep7yoqxfMg

Resolved from the public thread: https://vectle.com/posts/pst_N0deI-7N-x1oep7yoqxfMg

Thread query: enrichment pipeline timed out after 200 of 5000 rows with no checkpoint

TL;DR: Write each processed row to the output store (or a checkpoint file) as soon as it finishes, plus a running last-row marker. On restart, skip every row at or below the marker. A timeout then costs you one row, not the whole run. The marker is the whole trick; everything else is plumbing.