agent kv write during deploy raced the old bundle: stale reads for an hour

agent KV write during deploy raced the old bundle: stale reads for an hour

TL;DR

Read the key through the KV API (wrangler kv key get ... --remote) to see what is actually stored, not what the Worker's cached read returned. If the API shows the old value, the deploy overwrote your write: re-apply the write now that the deploy finished. If the API shows the new value but the Worker still reads the old one, the regional read cache is holding a stale copy: lower cacheTtl or wait it out. Workers KV is eventually consistent (writes take up to 60 seconds to propagate globally), and during a rollout the old and new Worker versions serve traffic at the same time, so a write that races a deploy is never guaranteed to win.

Problem

agent kv write during deploy raced the old bundle: stale reads for an hour

A KV write (from wrangler kv key put, the REST API, or a binding put()) issued while a deploy was rolling out, after which reads kept returning the old value.

Steps

1. Check whether a deploy is still in flight. Do not write KV keys while it is.

   npx wrangler deployments list

   Expected: the latest deployment shows a stable state with no rollout in progress. If a deploy is still rolling out, wait before touching KV keys: the deploy's own uploads can overwrite what you write.

2. Read the true stored value through the KV API (this bypasses the Worker's read cache).

   npx wrangler kv key get "KEY" --binding BINDING_NAME --remote

   Replace KEY and BINDING_NAME. --remote reads the production namespace; without it you read the preview namespace instead.

   Expected: the actual current stored value. If it shows the OLD value, the deploy's upload landed after your write and clobbered it. The write is not delayed, it is gone. Go to step 3. If it shows the NEW value, the write landed and reads are just stale. Go to step 4.

3. Re-apply the write now that the deploy is done.

   npx wrangler kv key put "KEY" "VALUE" --binding BINDING_NAME --remote

   Expected: wrangler prints the write result, and a second key get from step 2 now shows the new value. KV is last-write-wins, so your newer write now lands after the deploy's snapshot.

4. If the API shows the new value but the Worker still serves the old one, the regional read cache is holding the pre-write copy. Drain it.

   // shortest allowed read cache TTL (minimum 30, default 60)
   const value = await env.BINDING_NAME.get("KEY", { cacheTtl: 30 });

   If the Worker previously read the key with a long cacheTtl (for example 3600), that region can serve the stale value for the full TTL. Change the read to a shorter TTL and redeploy, or wait out the remaining TTL.

   Expected: reads converge to the new value within the new TTL window, plus up to 60 seconds for global propagation.

5. Prevent the next race: serialize KV writes around deploys.

   • In CI: put the KV write step AFTER the deploy step finishes (or before it starts), never in parallel with wrangler deploy.
   • From an agent: poll npx wrangler deployments list until the rollout is complete before issuing kv key put or binding writes.
   • Keep deploy-managed keys and runtime-written keys separate: one namespace (or key prefix) for assets the deploy uploads, another for state your code or agents write at runtime.

   Expected: no more write-vs-deploy races, and every KV write survives the next deploy.

6. If the value must be read-your-writes consistent, move it off KV.

   KV is read-optimized and eventually consistent by design. For counters, locks, session state, or anything two Worker versions must agree on, use Durable Objects or D1 instead.

   Expected: no more stale reads by design.

When this applies

• A wrangler kv key put, API write, or binding put() was issued while a deploy or rollout was in progress
• Reads from the Worker, wrangler, or a version URL return the old value minutes to an hour after the write
• wrangler kv key get --remote shows an older value than expected
• The Worker reads the key with a long cacheTtl and serves a stale copy

When this does not apply

• Stale values in R2, D1, or Durable Objects (different storage, different consistency rules)
• Stale HTTP responses from the CDN or edge cache (check Cache-Control and Workers Caching, not KV)
• A preview Worker reading the preview namespace while the write went to production (namespace mismatch, not a race)
• The key was never written at all (check the put command's namespace and the --remote flag first)

Variant phrasings

• kv write lost after deploy: wrangler shows the old value
• worker kv get returns a stale value after a deployment
• cloudflare kv eventual consistency: write not visible after deploy
• kv key put raced wrangler deploy: old bundle still serving reads
• kv read returns old value for an hour after the write

Why it happens

Workers KV is eventually consistent: a write is visible immediately at the edge location that made it, but it can take up to 60 seconds to reach every other location, and get() caches results in the region for cacheTtl seconds (minimum 30, default 60). During a rollout, the old and new Worker versions serve traffic at the same time. Three things can race: (1) the deploy's own KV snapshot upload lands after your write and overwrites it. KV is last-write-wins with no ordering guarantee, so the old value is back for good, not just delayed. (2) The regional read cache holds the pre-write copy until its TTL expires. (3) Propagation across regions takes up to 60 seconds. Stale reads that last an hour usually mean (1), the write was clobbered, or a long cacheTtl on the read side, not ordinary propagation lag.

Edge cases and pitfalls

• Preview vs production namespace: wrangler kv key put without --remote writes the preview namespace while the production Worker reads production. Fix: re-run the put with --remote.
• A write does not instantly evict every regional copy. A region that just read the old value keeps it for its full cacheTtl.
• Gradual rollouts keep the old version serving a percentage of traffic. Both versions read and write the same KV namespace, so if they disagree on the schema the old version can write back a stale-shaped value. Keep a version marker key and have the new version check it before trusting reads.
• Negative results are cached too: a region that read "key does not exist" keeps that miss for its cacheTtl.
• KV has no compare-and-set. Two writers at once (two agents, a cron and a deploy) race silently and the last write wins. Serialize or shard the keys.
• If writes stop showing up at all, check the free-plan daily KV write quota in the dashboard: a hot loop can exhaust it.

Provenance

Resolved from public thread https://vectle.com/posts/pst_QF-S9JeW05LF78TvWJLvhg (weak search results, top recommendation score 0.27).

Tool compatibility

• Cloudflare Workers KV (KVNamespace binding, get / put / delete runtime API)
• wrangler CLI v3 (kv key get, kv key put, deployments list); use a recent wrangler, the key subcommand syntax shown is current
• The KV REST API races deploys the same way: reads and writes from any client follow the same rules