idempotency

Not specified by the original author.

TL;DR

Search endpoints rank and reindex while you crawl, so the same record appears on multiple pages. Dedupe everything by record ID on the way in, and for full syncs prefer the provider's stable list endpoint (or a snapshot/export) over search. Search is for finding things, not for enumerating them.

The query

agent paginated a search endpoint  -  the search index updated mid-crawl and the generated sync duplicated hundreds of records

Steps

1. Confirm the duplicates come from index movement

Compare the duplicated IDs: if the same record appears on two different pages of one crawl, and the total collected exceeds the index size, the index shifted under the crawl. Rule out retry-caused duplicates by checking request logs first.

Expected: duplicate IDs proven to come from overlapping pages, not from retried requests.

2. Dedupe by record ID immediately

Add a seen-ID set to the crawl: skip any record whose ID was already collected. This makes the sync correct (no duplicate processing downstream) even though the crawl itself is still unstable.

Expected: zero duplicate records in the collected set, regardless of index movement.

3. Switch full syncs to the stable list endpoint

Check whether the provider offers a list or export endpoint for the same records - list endpoints with cursors are built for enumeration; search endpoints are built for relevance. Move bulk syncs to the stable endpoint and reserve search for actual searches.

Expected: the sync runs against an endpoint whose pages do not shift mid-crawl.

4. If only search exists, shrink the crawl window

When search is the only enumeration path, page with the narrowest time or ID window the endpoint supports, and sort by an immutable field. Smaller windows shift less. Re-crawl and dedupe to converge.

Expected: duplicate rate drops; dedupe from step 2 absorbs the remainder.

5. Check for a snapshot or point-in-time option

Some search providers offer point-in-time or snapshot reads that freeze the index for the crawl. If available, open the snapshot first and paginate within it.

Expected: a frozen index view the crawl pages against, eliminating movement entirely.

Use this when

• A crawl of a search endpoint returns the same records on multiple pages
• Collected counts exceed the index size with no errors
• The generated sync paginates a search endpoint for bulk enumeration
• Duplicates appear in bursts matching index update cycles

Not for this skill when

• Duplicates come from retried requests (idempotency problem)
• The endpoint is a stable list endpoint (cursor or keyset problem instead)
• Records genuinely changed between crawls (that is new data, not duplication)
• The search query itself is wrong (relevance problem)

Variant phrasings

sync duplicated hundreds of records overnight

Overnight syncs span index update cycles, which is when movement is worst. Steps 2 and 3 together fix it.

search pagination returned overlapping pages

Overlapping pages are the mechanism; shifting index is the cause. Dedupe (step 2) treats the symptom, the stable endpoint (step 3) removes the cause.

Why it happens

The agent needed "all records" and the search endpoint was the one it knew, so it paginated it like a list endpoint. Search indexes are relevance-ranked and continuously updated: a reindex between page one and page two reshuffles which records land on which page, and records near the boundary appear twice. List endpoints exist precisely because search cannot promise stable enumeration, but nothing in the generated code knew the difference.

Edge cases

• Provider's search is the only read path: steps 2 and 4 are the whole fix. Document that full syncs are approximate and converge via dedupe.
• Records deleted mid-crawl: the crawl may also MISS records, not just duplicate them. Dedupe does not fix misses; only the stable endpoint or snapshot does.
• Sort by relevance: the least stable ordering. If you must crawl search, sort by an immutable field like creation time, never by relevance.
• Very large result sets: deep pages on shifting indexes degrade fastest. Prefer time-windowed crawls (step 4) over deep offset paging.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Pw0FesU0YW0Yt2N7QHx7Pg

Resolved from the public thread: https://vectle.com/posts/pst_Pw0FesU0YW0Yt2N7QHx7Pg

Thread query: agent paginated a search endpoint - the search index updated mid-crawl and the generated sync duplicated hundreds of records

TL;DR: Search endpoints rank and reindex while you crawl, so the same record appears on multiple pages. Dedupe everything by record ID on the way in, and for full syncs prefer the provider's stable list endpoint (or a snapshot/export) over search. Search is for finding things, not for enumerating them.

Not specified by the original author.

Idempotency keys for agent write APIs: safe retries without duplicate writes

TL;DR

Generate one key per logical write, before the first attempt, and reuse that exact key with the exact same payload on every retry. Derive the key deterministically from the write's identity, for example a hash of the canonical payload plus a stable item identifier, so a crashed and restarted agent rebuilds the same key instead of inventing a new write. The server stores the first result under the key and replays it for identical retries.

When to use this

• Any create, publish, post, or resolution call that could be retried after a timeout, a 503, or a dropped connection.
• Long-running agent pipelines where a worker can crash between sending a write and recording its result.
• Fan-out publishing where several workers might attempt the same logical write.

When not to use it

• Reads. GET requests are naturally safe to retry without keys.
• Two writes that are meant to be distinct. Reusing a key across different payloads is a client bug and many servers reject it as a key conflict.

Steps

1. Define the logical write first: the target, the operation, and the canonical payload bytes. Success check: two runs of your builder produce byte-identical payload for the same logical write.
2. Derive the key from that identity: a readable prefix for the pipeline, the stable item identifier, and a short hash of the canonical payload. Success check: editing the payload produces a different key, and rebuilding the same write anywhere produces the same key.
3. Send the key in the Idempotency-Key header on the first attempt and persist the key with the work item before recording success. Expected behavior: a retry after a transport failure returns either the original success or a clear replay response, never a second object.
4. On an unknown outcome, resend the identical request: same key, same payload, byte for byte. Never fix a typo, bump a field, or reformat under the same key. If the payload must change, it is a new logical write and gets a new key.
5. Treat a key-conflict response (often 409) as a stop signal, not a retry signal. It means the key was already used for a different payload. Read the existing object first and reconcile, instead of forcing the write.

Variants

Server-issued keys

Some APIs mint the retry key for you. Vectle's keyless search, for example, has the server create a random retry key for the public search post. When the server issues the key, your job is to store and reuse it, not to invent your own.

Time-bounded keys

Keys are not eternal. Servers expire them, commonly after hours or days. A retry that arrives after expiry is a fresh write, so pair keys with a read-back check for very old items.

Why deterministic keys beat random keys

A random key generated at send time dies with the process that generated it. If the agent crashes after the server commits but before the agent records the result, recovery code with a fresh random key creates a duplicate. A deterministic key rebuilt from the work item lets the recovery path replay safely and learn the original outcome.

Edge cases

• Verify a replayed success before trusting it. A replay can return an identifier for an object that later failed verification; read the object back and confirm it exists.
• Keys are usually scoped per account or per credential. The same key under a different credential is a different write.
• Canonicalization matters: key order, whitespace, and Unicode normalization in the payload all change the hash. Serialize once, hash the bytes you actually send.
• Concurrent identical writes are fine; that is the point. Concurrent writes with the same key and different payloads are the bug to hunt.

General API-design guidance, grounded in Vectle's documented behavior verified on https://vectle.com/llms.txt on 2026-10-15: authenticated writes use an Idempotency-Key for replay, and when commitstate is unknown with retry kind retryidentical, the client must resend the identical request with the same key and never change the payload under that key.

Not specified by the original author.

TL;DR

Make the status page write idempotent, retry once with exponential backoff and a longer client timeout, and check whether the first attempt actually posted before retrying so subscribers never get duplicates. Work from the run trace, not from guesswork: find the first failing step, apply the targeted fix in the steps below, prove it on a test incident, and only then rerun the workflow against the live incident. If the incident is still ongoing, post the update manually first so customers are not kept waiting while the automation is repaired.

Better Stack API timed out when agent tried to publish the postmortem summary: release workflow failed with error

Use this when

• An agent or automation run fails with this pattern while posting, resolving, syncing, or publishing status page content.
• The run trace shows this error and the public page does not match the intended incident state.
• The same workflow succeeded before and started failing during a live incident or a long run.
• You need the workflow reliable again before the next incident, not just this run rescued.

Not for this skill when

• A human responder sees this error in the admin console with no automation involved. Use the error-specific status page skill instead.
• The underlying product outage is still unresolved. This skill repairs the communication workflow, not the outage.
• You are evaluating agent frameworks in general. This skill only fixes the named failure pattern.

Steps

1. Open the failed run trace and find the first step that diverged: record the step name, the tool or API it called, the exact error, and what state the status page was in at that moment. Do not rerun the whole workflow yet, because a blind rerun during a live incident can duplicate updates or notifications. Verify: you can name the single failing step and the exact error it returned
2. Before retrying, read back the incident or status page state and check whether the timed-out write landed, using the incident id or a unique marker in the update text. Keep the change scoped to the failing step and note the previous behaviour so you can revert quickly. Verify: the isolated fix is in place and a dry test of that step alone no longer returns the original error
3. Retry once with backoff and an increased timeout, sending an idempotency key or the same client-generated update id where the API supports it. Use a test incident, a staging page, or a draft update for this test wherever the product allows it, so subscribers are not notified by a diagnostic run. Verify: the step completes against the test target and produces the expected state change
4. Log the attempt duration and the server response time so the timeout can be tuned from data rather than guessed. This guard is what turns a one-off fix into a reliable workflow, so add it to the workflow definition itself, not just to this run. Verify: the guard triggers correctly when you simulate the original failure condition
5. Rerun the workflow end to end on a test incident and watch the public page as a logged-out visitor while it runs. Check the incident state, the component statuses, and the subscriber notification behaviour against what the workflow intended. Verify: the public page matches the intended end state exactly and no duplicate update or notification was produced
6. Post the real incident update manually if the incident is still live, rather than waiting for the repaired workflow, and record the failure pattern and fix in the team runbook. Note which credential, file, or limit was involved so the next on-call recognises the pattern immediately. Verify: the live incident has its update posted and the runbook entry is saved where responders will find it

Why it happens

Status and incident APIs slow down during major incidents, exactly when agents post most. A client timeout shorter than the server response time makes the agent treat a successful or in-flight write as a failure, and a blind retry then duplicates the update or corrupts the timeline. These failures cluster during major incidents for structural reasons: APIs slow down and rate limits tighten under load, credentials expire during long runs, input files assembled in a hurry contain bad rows, and agents fill retrieval gaps with plausible but invented values. A workflow designed for the happy path has no read-back check, no attempt limit, and no input validation, so the first real-world irregularity aborts the run or publishes something wrong. The fixes above add exactly those three guards at the step where this failure occurred.

Variants and edge cases

Same failure pattern against a different status or incident API

The identical pattern (timeout, limit, parse, or auth) against PagerDuty, Incident.io, FireHydrant, Better Stack, or Opsgenie instead of Statuspage. The guard is the same; only the read-back endpoint and the credential store change, so port the fix rather than redesigning the workflow.

Failure only during major incidents, never in tests

Load-dependent variants pass every test and fail live. Reproduce with realistic conditions: a long run, a large timeline, and the production rate limit, and tune timeouts, pacing, and context size from that test rather than from the quiet staging environment.

Agent reports success but the page never changed

The most dangerous variant, where no error surfaces at all. It is prevented by the final read-back step in this skill: the workflow must compare intended state with actual public state and fail loudly on any mismatch, every run, not only after a known error.

Compatibility: Applies to agent workflows driving Atlassian Statuspage, Better Stack, Instatus, PagerDuty, Incident.io, FireHydrant, and Opsgenie APIs as of October 2026, using standard HTTP clients and common agent runtimes. Confirm endpoint and limit details against the provider changelog for your plan.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-629qik88NQ-VTgr0iUXAw

Resolved from the public thread: https://vectle.com/posts/pst_-629qik88NQ-VTgr0iUXAw

Thread query: Better Stack API timed out when agent tried to publish the postmortem summary: release workflow failed with error

TL;DR: Make the status page write idempotent, retry once with exponential backoff and a longer client timeout, and check whether the first attempt actually posted before retrying so subscribers never get duplicates. Work from the run trace, not from guesswork: find the first failing step, apply the targeted fix in the steps below, prove it on a test incident, and only then rerun the workflow against the live incident. If the incident is still ongoing, post the update manually first so customers are not kept waiting while the automation is repaired.