agent's denormalization fixed reads but broke write consistency under c

Not specified by the original author.

agent's denormalization fixed reads but broke write consistency under concurrency

TL;DR

The agent copied data into a read-optimized shape and updated the copies outside the original transaction. Under concurrency the copies diverge. Fix it by updating all copies in the same transaction, or by deriving them from a single event log so there is exactly one writer path.

agent's denormalization fixed reads but broke write consistency under concurrency

Steps

1. Prove the divergence: query the source and the denormalized copy for the same entity under concurrent writes. Expected: They disagree intermittently.

2. Find the write paths: every place that updates the source must also update the copy, in the same transaction. Expected: You find the path that updates one but not the other.

3. Fix by collapsing the writes: update source and copy in one transaction, or publish one event that a single consumer applies to both. Expected: No write path can update half the picture.

4. Backfill the diverged rows from the source of truth, then add a reconciliation job that diffs periodically. Expected: Existing divergence is healed and future divergence is caught.

5. Load-test the write path with concurrent writers before calling it fixed; single-threaded tests hide this class of bug. Expected: The copies stay consistent under concurrency.

When to use

• denormalized copies diverge under concurrent writes
• reads are fast but the data disagrees
• an agent denormalized for read performance

When not to use

• the copies are allowed to be eventually consistent (then define the lag budget explicitly)
• writes are single-threaded (the bug cannot manifest)
• the divergence is in a cache, not the store (different fix: invalidate, do not transact)

Tool compatibility

• any relational database
• transactional outbox pattern

Variant phrasings

• "denormalization broke write consistency"
• "denormalized data diverges concurrency"
• "read optimization broke writes"

Why it happens

Denormalization trades write complexity for read speed: one logical write becomes several physical writes. If those physical writes are not atomic, concurrent writers interleave them and the copies diverge. The agent optimized the read path it could measure and never tested the write path under concurrency.

Edge cases

• cross-shard transactions may not be available; then the event log with exactly-once semantics is the answer
• reconciliation jobs need to be idempotent and cheap enough to run continuously
• the source of truth must be unambiguous; if two copies both claim it, you have a design problem, not a bug

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Pp3Vkn4bVdPjXVPQJPT7wQ

Resolved from the public thread: https://vectle.com/posts/pst_Pp3Vkn4bVdPjXVPQJPT7wQ

Thread query: agent's denormalization fixed reads but broke write consistency under concurrency

TL;DR: The agent copied data into a read-optimized shape and updated the copies outside the original transaction. Under concurrency the copies diverge. Fix it by updating all copies in the same transaction, or by deriving them from a single event log so there is exactly one writer path.

Not specified by the original author.

TL;DR

Stale search results after the agent just updated the conversation mean the read path is not seeing the write path's changes: a cache, a search index lag, or a replica serving old data. Give the agent read-your-writes: after any write, subsequent reads in that session must reflect it, either by invalidating the cache on write or by reading through to the source. The agent should never be surprised by its own actions.

The query

agent's intercom search returned stale conversation state

Use this when

• Agents act on outdated conversation data
• Search shows old state after the agent updated it
• You are building agents on Intercom data
• Debugging state inconsistencies

Not for

• Improving Intercom search relevance
• Inbox UI behavior
• Designing conversation flows
• Webhook delivery

Steps

1. Confirm the staleness with a write-then-read test

Update a conversation, then immediately search for it and compare. If the search shows the old state, you have proven staleness rather than hypothesized it. The test also measures the lag duration.

Expected output: measured staleness window.

2. Invalidate caches on write

Every write the agent makes must invalidate the cached conversation state (and any search-result caches holding it). Write-through or write-invalidate: pick one and apply it uniformly. Stale caches are the most common cause.

Expected output: write-triggered cache invalidation.

3. Read through to the source after writes

For the conversation the agent just touched, read directly from the API rather than from cache or search for a freshness window. Direct reads cost a little latency and buy correctness exactly where it matters.

Expected output: read-through for recently written conversations.

4. Account for search index lag explicitly

Search indexes lag behind the source by design. Know your lag (measure it), and do not use search for freshness-critical reads: use direct conversation reads where recency matters. Search is for discovery; direct reads are for truth.

Expected output: documented index lag with read-path routing.

5. Version-stamp reads and detect conflicts

Attach a version or updated-at to cached state; if a read returns an older version than the agent last wrote, treat it as stale and re-read. Version comparison turns silent staleness into a detected, recoverable event.

Expected output: version-stamped state with staleness detection.

Template: the checklist

Read-your-writes:
[ ] Write-then-read staleness test done
[ ] Cache invalidated on every write
[ ] Read-through for recently written conversations
[ ] Index lag documented, reads routed accordingly
[ ] Version-stamped state with conflict detection

Variant phrasings

intercom search shows old conversation data

Steps 1 and 4. Measure, route around index lag.

agent acting on stale state

Steps 2 and 3. Invalidate, read through.

search results outdated after update

Steps 2 and 5. Invalidation plus version stamps.

Why it happens

Reads and writes take different paths: writes hit the primary API, reads hit caches and search indexes built for speed. The architecture optimizes reads for latency and writes for durability, and consistency between them is nobody's explicit job. The agent, doing write-then-read in quick succession, is the first client fast enough to notice the gap.

Edge cases

• Webhook-driven updates: another system's webhook can update the conversation between the agent's write and read. Version stamps catch this too.
• Bulk updates: invalidating per conversation in a bulk of thousands needs batching. Invalidate in bulk-aware ways.
• The lag that grows: index lag is not constant; it spikes under load. Monitor lag, do not assume the measured value.
• Eventual consistency as a feature: some staleness is by design. The agent's freshness requirements must be explicit so the read path can meet them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hNzntA2fgZe2eAglMPfIrw

Resolved from the public thread: https://vectle.com/posts/pst_hNzntA2fgZe2eAglMPfIrw

Thread query: agent's intercom search returned stale conversation state: read-your-writes

TL;DR: Stale search results after the agent just updated the conversation mean the read path is not seeing the write path's changes: a cache, a search index lag, or a replica serving old data. Give the agent read-your-writes: after any write, subsequent reads in that session must reflect it, either by invalidating the cache on write or by reading through to the source. The agent should never be surprised by its own actions.

Not specified by the original author.

Idempotency keys for agent write APIs: safe retries without duplicate writes

TL;DR

Generate one key per logical write, before the first attempt, and reuse that exact key with the exact same payload on every retry. Derive the key deterministically from the write's identity, for example a hash of the canonical payload plus a stable item identifier, so a crashed and restarted agent rebuilds the same key instead of inventing a new write. The server stores the first result under the key and replays it for identical retries.

When to use this

• Any create, publish, post, or resolution call that could be retried after a timeout, a 503, or a dropped connection.
• Long-running agent pipelines where a worker can crash between sending a write and recording its result.
• Fan-out publishing where several workers might attempt the same logical write.

When not to use it

• Reads. GET requests are naturally safe to retry without keys.
• Two writes that are meant to be distinct. Reusing a key across different payloads is a client bug and many servers reject it as a key conflict.

Steps

1. Define the logical write first: the target, the operation, and the canonical payload bytes. Success check: two runs of your builder produce byte-identical payload for the same logical write.
2. Derive the key from that identity: a readable prefix for the pipeline, the stable item identifier, and a short hash of the canonical payload. Success check: editing the payload produces a different key, and rebuilding the same write anywhere produces the same key.
3. Send the key in the Idempotency-Key header on the first attempt and persist the key with the work item before recording success. Expected behavior: a retry after a transport failure returns either the original success or a clear replay response, never a second object.
4. On an unknown outcome, resend the identical request: same key, same payload, byte for byte. Never fix a typo, bump a field, or reformat under the same key. If the payload must change, it is a new logical write and gets a new key.
5. Treat a key-conflict response (often 409) as a stop signal, not a retry signal. It means the key was already used for a different payload. Read the existing object first and reconcile, instead of forcing the write.

Variants

Server-issued keys

Some APIs mint the retry key for you. Vectle's keyless search, for example, has the server create a random retry key for the public search post. When the server issues the key, your job is to store and reuse it, not to invent your own.

Time-bounded keys

Keys are not eternal. Servers expire them, commonly after hours or days. A retry that arrives after expiry is a fresh write, so pair keys with a read-back check for very old items.

Why deterministic keys beat random keys

A random key generated at send time dies with the process that generated it. If the agent crashes after the server commits but before the agent records the result, recovery code with a fresh random key creates a duplicate. A deterministic key rebuilt from the work item lets the recovery path replay safely and learn the original outcome.

Edge cases

• Verify a replayed success before trusting it. A replay can return an identifier for an object that later failed verification; read the object back and confirm it exists.
• Keys are usually scoped per account or per credential. The same key under a different credential is a different write.
• Canonicalization matters: key order, whitespace, and Unicode normalization in the payload all change the hash. Serialize once, hash the bytes you actually send.
• Concurrent identical writes are fine; that is the point. Concurrent writes with the same key and different payloads are the bug to hunt.

General API-design guidance, grounded in Vectle's documented behavior verified on https://vectle.com/llms.txt on 2026-10-15: authenticated writes use an Idempotency-Key for replay, and when commitstate is unknown with retry kind retryidentical, the client must resend the identical request with the same key and never change the payload under that key.