## TL;DR
Write each processed row to the output store (or a checkpoint file) as soon as it finishes, plus a running last-row marker. On restart, skip every row at or below the marker. A timeout then costs you one row, not the whole run. The marker is the whole trick; everything else is plumbing.

```text
enrichment pipeline timed out after 200 of 5000 rows with no checkpoint
```

## Use this when
- a pipeline dies partway and restarts from row one
- you re-enrich rows that were already done
- timeouts or restarts wipe in-memory progress

## Not for this skill when
- the enrichment API itself is down (fix the call first)
- you need to pick an enrichment vendor
- rows fail because the input data is bad (clean the input)

## Steps
1. Add a processed_at column or a checkpoint file holding the last finished row id. Verify: after 5 rows the marker shows row 5.
2. Process rows in small batches (25 to 50), committing the checkpoint after each batch. Verify: kill the process mid-run and the marker is within one batch of reality.
3. On startup, query for rows without processed_at and begin there. Verify: a second run touches zero already-done rows.
4. Make each row operation idempotent: upsert by row id, never blind insert. Verify: running the same batch twice changes nothing.
5. Set a per-row timeout with a retry budget of 3 on 5xx and timeouts only, then mark the row failed and move on. Verify: one bad row cannot stall the run.

### Tool/version compatibility
Any stack: Python scripts, Airflow, or plain cron. Works with Postgres, SQLite, or a flat JSONL checkpoint file.

## Why it happens
The pipeline holds progress in memory and writes results only at the end. Any interruption discards everything, so the default recovery is a full restart.

## Edge cases / pitfalls
- A checkpoint write that fails silently is worse than none: verify the write landed.
- Clock skew can confuse timestamp markers; prefer monotonic row ids.
- Marking rows failed still counts as progress: a poison row should not be retried forever.

## Variant phrasings
### Pipeline kept reprocessing the same rows after every timeout
Short version: no checkpoint between runs.
### Long enrichment job never finishes because it restarts
Short version: progress not persisted per row.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_N0deI-7N-x1oep7yoqxfMg