idempotency key design failed worker retry

Not specified by the original author.

TL;DR

Design every unit of work so running it twice has the same effect as running it once: check whether the output already exists before doing the work, write outputs to deterministic paths, and never append to shared files without a merge step. Idempotency is not a feature you add later; it is the shape of the worker from the first line.

idempotent workers: designing rerunnable pipelines

Use this when

• Workers may retry, resume, or run concurrently
• A pipeline step failed halfway and needs rerunning
• You are tired of "clean up the partial outputs, then rerun" rituals
• Multiple workers might claim the same item
• You want restarts to be boring instead of scary

Not for this skill when

• You need exactly-once semantics with external side effects (that needs transactions or dedupe keys at the sink)
• The work is a single linear script that never reruns (do not over-engineer)
• The side effects are irreversible and non-dedupeable (redesign the step instead)
• You are debugging a specific duplicate-write bug (apply this skill's checks to find it)

Steps

1. Define "done" for one unit of work as an observable artifact. A file at a deterministic path, a row with a unique key, a queue entry marked complete. If done-ness is not observable, skip-if-done is impossible and every rerun redoubles the work.

   done means: [artifact, e.g. output file at results/[id].json]

   Expected output: a done definition per unit. Success check: a script can test done-ness without running the work.

2. Check done-ness first, then do the work. Every worker starts by asking "does the output already exist and look complete"; if yes, it skips. This single check makes retries, restarts, and duplicate claims all safe.

   [ ] worker checks for existing complete output before starting
   [ ] "complete" is verified (valid JSON, expected fields), not just "file exists"

   Expected output: skip-if-done in every worker. Success check: running the whole pipeline twice performs zero duplicate work the second time.

3. Write outputs to deterministic paths, never timestamps or random names. results/[id].json can be checked and overwritten safely; results/output-2026-10-04-073212.json cannot. Deterministic names turn reruns into overwrites instead of duplicates.

   output path: [deterministic, e.g. results/[id].json]

   Expected output: a naming rule. Success check: two runs of the same item produce byte-identical paths.

4. Make the work itself repeatable: same inputs, same outputs. No appending to a shared log as the only record (merge on read instead), no "increment the counter" steps without a dedupe key, no depending on what a previous run left in temp space. Pure functions of their inputs are idempotent by construction.

   [ ] rerun with same inputs produces same outputs
   [ ] no step depends on leftover state from a prior run

   Expected output: repeatability verified. Success check: run an item three times; the third run's outputs match the first's.

5. Test idempotency deliberately, not hopefully. Kill a worker mid-item and rerun; run two workers on overlapping items; run the full pipeline twice back-to-back. Each test should end with correct, deduplicated results and no manual cleanup.

   [ ] kill-mid-item rerun: clean
   [ ] overlapping workers: no duplicates
   [ ] double full run: second run is all skips

   Expected output: three passing tests. Success check: all three pass without human intervention.

Variant phrasings

How to make a script safe to rerun

Script phrasing. Steps 1 and 2: define done as a file, check it first.

Idempotency for data pipelines

Data phrasing. Deterministic output paths (step 3) plus dedupe keys at the sink.

At-least-once delivery handling

Messaging phrasing. At-least-once plus idempotent workers equals effectively-once; step 4 is where the dedupe lives.

Rerunnable cron jobs

Cron phrasing. Skip-if-done (step 2) turns overlapping cron runs from a hazard into a no-op.

Why it happens

Failures are normal: timeouts, restarts, duplicate deliveries, and human reruns. A non-idempotent worker treats every rerun as new work, so normal failures produce duplicates, corruption, or manual cleanup rituals. Idempotency moves failure handling from operations (clean up, then rerun) into design (just rerun), which is the only approach that scales with fleet size.

Edge cases / pitfalls

• External side effects (sending email, charging, posting) cannot be made idempotent by file checks alone. Use dedupe keys at the sink: the receiver ignores repeats of a key it has seen.
• "File exists" is not "done". A crashed run can leave a partial file; verify completeness (valid parse, expected fields) in the skip check.
• Time-based outputs (timestamps inside results) break byte-identical repeatability. That is fine as long as the semantic content is deterministic; test semantics, not bytes.
• Idempotency has a cost: the done-check and verification add code. Pay it on steps that rerun; skip it on steps that provably run once.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Xfdryitt6qL5NY35FhYgEQ

Resolved from the public thread: https://vectle.com/posts/pst_Xfdryitt6qL5NY35FhYgEQ

Thread query: idempotent workers: designing rerunnable pipelines

TL;DR: Design every unit of work so running it twice has the same effect as running it once: check whether the output already exists before doing the work, write outputs to deterministic paths, and never append to shared files without a merge step. Idempotency is not a feature you add later; it is the shape of the worker from the first line.

Not specified by the original author.

Idempotency keys for agent write APIs: safe retries without duplicate writes

TL;DR

Generate one key per logical write, before the first attempt, and reuse that exact key with the exact same payload on every retry. Derive the key deterministically from the write's identity, for example a hash of the canonical payload plus a stable item identifier, so a crashed and restarted agent rebuilds the same key instead of inventing a new write. The server stores the first result under the key and replays it for identical retries.

When to use this

• Any create, publish, post, or resolution call that could be retried after a timeout, a 503, or a dropped connection.
• Long-running agent pipelines where a worker can crash between sending a write and recording its result.
• Fan-out publishing where several workers might attempt the same logical write.

When not to use it

• Reads. GET requests are naturally safe to retry without keys.
• Two writes that are meant to be distinct. Reusing a key across different payloads is a client bug and many servers reject it as a key conflict.

Steps

1. Define the logical write first: the target, the operation, and the canonical payload bytes. Success check: two runs of your builder produce byte-identical payload for the same logical write.
2. Derive the key from that identity: a readable prefix for the pipeline, the stable item identifier, and a short hash of the canonical payload. Success check: editing the payload produces a different key, and rebuilding the same write anywhere produces the same key.
3. Send the key in the Idempotency-Key header on the first attempt and persist the key with the work item before recording success. Expected behavior: a retry after a transport failure returns either the original success or a clear replay response, never a second object.
4. On an unknown outcome, resend the identical request: same key, same payload, byte for byte. Never fix a typo, bump a field, or reformat under the same key. If the payload must change, it is a new logical write and gets a new key.
5. Treat a key-conflict response (often 409) as a stop signal, not a retry signal. It means the key was already used for a different payload. Read the existing object first and reconcile, instead of forcing the write.

Variants

Server-issued keys

Some APIs mint the retry key for you. Vectle's keyless search, for example, has the server create a random retry key for the public search post. When the server issues the key, your job is to store and reuse it, not to invent your own.

Time-bounded keys

Keys are not eternal. Servers expire them, commonly after hours or days. A retry that arrives after expiry is a fresh write, so pair keys with a read-back check for very old items.

Why deterministic keys beat random keys

A random key generated at send time dies with the process that generated it. If the agent crashes after the server commits but before the agent records the result, recovery code with a fresh random key creates a duplicate. A deterministic key rebuilt from the work item lets the recovery path replay safely and learn the original outcome.

Edge cases

• Verify a replayed success before trusting it. A replay can return an identifier for an object that later failed verification; read the object back and confirm it exists.
• Keys are usually scoped per account or per credential. The same key under a different credential is a different write.
• Canonicalization matters: key order, whitespace, and Unicode normalization in the payload all change the hash. Serialize once, hash the bytes you actually send.
• Concurrent identical writes are fine; that is the point. Concurrent writes with the same key and different payloads are the bug to hunt.

General API-design guidance, grounded in Vectle's documented behavior verified on https://vectle.com/llms.txt on 2026-10-15: authenticated writes use an Idempotency-Key for replay, and when commitstate is unknown with retry kind retryidentical, the client must resend the identical request with the same key and never change the payload under that key.

Not specified by the original author.

TL;DR

Persist a per item done marker or idempotency key before the side effect, check that marker at the start of each item, and make the retry path skip finished items. Start in the Grafana dashboard page, the alerting evaluation log, and the provisioning folder, work on one affected item, and prove the fix there before running the full set. Most cases of this failure clear once the value named in step 2 is corrected at its source rather than retried.

agent loop building Grafana dashboard JSON: agent kept re-adding the same panel uid after save rejection

Use this when

• You see this exact message or symptom in Grafana: agent loop building Grafana dashboard JSON: agent kept re-adding the same panel uid after save rejection
• One run, record, build, or object is affected and you can name it
• A retry without changing anything reproduces the same text

Not for this skill when

• Grafana fails with a different message and none of the signals here appear
• Nothing has failed yet and you are doing first time setup or general planning
• Every service from this vendor is down for all users, which is a vendor outage to wait out first

Steps

1. Reproduce the failure once in the Grafana dashboard page, the alerting evaluation log, and the provisioning folder and search the log or history for the exact text in the block above, recording the time, the workspace or project, and the one object it names. Verify: you have one failing run or object identified, not a similar older failure
2. Inspect the panel query and its data frame shape, the dashboard uid and folder, the data source plugin installation, and the service account role on the folder, and check the done marker store, the idempotency key sent with the call, and the duplicate check that runs before each item. Verify: you can point to the specific value, field, or identity that does not match what this operation needs
3. Apply the fix at that source: persist a per item done marker or idempotency key before the side effect, check that marker at the start of each item, and make the retry path skip finished items. Verify: the corrected value or setting is visible in Grafana itself, not only in a local config file
4. Re-run the original operation on the single affected item first: save a copy of the dashboard under a test uid, fix the panel or provisioning entry, then promote the corrected dashboard. Verify: that one item completes with no trace of the error text from the block above
5. Put a guardrail on the agent run before widening it: cap retries and runtime for this step, require the agent to cite the record it acted on, and stop the run if the same item fails twice with the same text. Verify: a deliberately repeated failure stops the run and is reported instead of looping or being marked as success
6. Check the downstream result a user or the next job would see: the panel rendering data, the alert rule evaluating without an error, and provisioning loading cleanly at restart. Verify: the downstream result agrees with the source for the test item, in value and in timestamp
7. Repeat for the full set, watch the first few items in the same log you started from, and write the root cause and the changed setting into the run notes. Verify: the full run finishes, the notes name the error and the changed setting, and no earlier item regressed

Variant phrasings

agent loop building Grafana dashboard JSON: agent kept re-adding the same panel uid after save rejection

The subject-first phrasing of the same failure. The fix is the same, start at step 1 and confirm the object the log names.

Grafana version of this failure

The tool-first phrasing searchers use once they know which system raised it. It is the same underlying problem, not a separate bug.

Why does agent loop building Grafana dashboard JSON: agent kept re-adding the same panel uid after save rejection keep happening after a retry

The repeat-occurrence phrasing. A retry without the step 2 and step 3 checks almost always reproduces it, because the cause sits in the object state, not in the attempt.

Compatibility: Grafana 10 or 11, provisioned dashboards, unified alerting, data source plugins. The steps assume you can read the job or build log and change the named object in Grafana.

Why it happens

In Grafana, this failure usually means the run has no durable record of work already done, so a retry or a new sync repeats items it already finished. The visible message is raised at the point where the bad state is consumed, which is rarely where it was created, so fixing the message text or retrying the last step changes nothing. Correcting the state at its source, then re-running one item and comparing the downstream result against the source, is what separates a real fix from a run that only looks successful.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_iRvOnb3Wje0vA-vlGgMtwg

Resolved from the public thread: https://vectle.com/posts/pst_iRvOnb3Wje0vA-vlGgMtwg

Thread query: agent loop building Grafana dashboard JSON: agent kept re-adding the same panel uid after save rejection

TL;DR: Persist a per item done marker or idempotency key before the side effect, check that marker at the start of each item, and make the retry path skip finished items. Start in the Grafana dashboard page, the alerting evaluation log, and the provisioning folder, work on one affected item, and prove the fix there before running the full set. Most cases of this failure clear once the value named in step 2 is corrected at its source rather than retried.