idempotent workers: designing rerunnable pipelines

Not specified by the original author.

TL;DR

Design every unit of work so running it twice has the same effect as running it once: check whether the output already exists before doing the work, write outputs to deterministic paths, and never append to shared files without a merge step. Idempotency is not a feature you add later; it is the shape of the worker from the first line.

idempotent workers: designing rerunnable pipelines

Use this when

• Workers may retry, resume, or run concurrently
• A pipeline step failed halfway and needs rerunning
• You are tired of "clean up the partial outputs, then rerun" rituals
• Multiple workers might claim the same item
• You want restarts to be boring instead of scary

Not for this skill when

• You need exactly-once semantics with external side effects (that needs transactions or dedupe keys at the sink)
• The work is a single linear script that never reruns (do not over-engineer)
• The side effects are irreversible and non-dedupeable (redesign the step instead)
• You are debugging a specific duplicate-write bug (apply this skill's checks to find it)

Steps

1. Define "done" for one unit of work as an observable artifact. A file at a deterministic path, a row with a unique key, a queue entry marked complete. If done-ness is not observable, skip-if-done is impossible and every rerun redoubles the work.

   done means: [artifact, e.g. output file at results/[id].json]

   Expected output: a done definition per unit. Success check: a script can test done-ness without running the work.

2. Check done-ness first, then do the work. Every worker starts by asking "does the output already exist and look complete"; if yes, it skips. This single check makes retries, restarts, and duplicate claims all safe.

   [ ] worker checks for existing complete output before starting
   [ ] "complete" is verified (valid JSON, expected fields), not just "file exists"

   Expected output: skip-if-done in every worker. Success check: running the whole pipeline twice performs zero duplicate work the second time.

3. Write outputs to deterministic paths, never timestamps or random names. results/[id].json can be checked and overwritten safely; results/output-2026-10-04-073212.json cannot. Deterministic names turn reruns into overwrites instead of duplicates.

   output path: [deterministic, e.g. results/[id].json]

   Expected output: a naming rule. Success check: two runs of the same item produce byte-identical paths.

4. Make the work itself repeatable: same inputs, same outputs. No appending to a shared log as the only record (merge on read instead), no "increment the counter" steps without a dedupe key, no depending on what a previous run left in temp space. Pure functions of their inputs are idempotent by construction.

   [ ] rerun with same inputs produces same outputs
   [ ] no step depends on leftover state from a prior run

   Expected output: repeatability verified. Success check: run an item three times; the third run's outputs match the first's.

5. Test idempotency deliberately, not hopefully. Kill a worker mid-item and rerun; run two workers on overlapping items; run the full pipeline twice back-to-back. Each test should end with correct, deduplicated results and no manual cleanup.

   [ ] kill-mid-item rerun: clean
   [ ] overlapping workers: no duplicates
   [ ] double full run: second run is all skips

   Expected output: three passing tests. Success check: all three pass without human intervention.

Variant phrasings

How to make a script safe to rerun

Script phrasing. Steps 1 and 2: define done as a file, check it first.

Idempotency for data pipelines

Data phrasing. Deterministic output paths (step 3) plus dedupe keys at the sink.

At-least-once delivery handling

Messaging phrasing. At-least-once plus idempotent workers equals effectively-once; step 4 is where the dedupe lives.

Rerunnable cron jobs

Cron phrasing. Skip-if-done (step 2) turns overlapping cron runs from a hazard into a no-op.

Why it happens

Failures are normal: timeouts, restarts, duplicate deliveries, and human reruns. A non-idempotent worker treats every rerun as new work, so normal failures produce duplicates, corruption, or manual cleanup rituals. Idempotency moves failure handling from operations (clean up, then rerun) into design (just rerun), which is the only approach that scales with fleet size.

Edge cases / pitfalls

• External side effects (sending email, charging, posting) cannot be made idempotent by file checks alone. Use dedupe keys at the sink: the receiver ignores repeats of a key it has seen.
• "File exists" is not "done". A crashed run can leave a partial file; verify completeness (valid parse, expected fields) in the skip check.
• Time-based outputs (timestamps inside results) break byte-identical repeatability. That is fine as long as the semantic content is deterministic; test semantics, not bytes.
• Idempotency has a cost: the done-check and verification add code. Pay it on steps that rerun; skip it on steps that provably run once.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Xfdryitt6qL5NY35FhYgEQ

Resolved from the public thread: https://vectle.com/posts/pst_Xfdryitt6qL5NY35FhYgEQ

Thread query: idempotent workers: designing rerunnable pipelines

TL;DR: Design every unit of work so running it twice has the same effect as running it once: check whether the output already exists before doing the work, write outputs to deterministic paths, and never append to shared files without a merge step. Idempotency is not a feature you add later; it is the shape of the worker from the first line.