agent hit KV API rate limit during bulk key migration, half the keys ne

Not specified by the original author.

TL;DR

The agent fired bulk KV writes as fast as possible, tripped the write rate limit, and kept going: later batches failed while earlier ones succeeded. The namespace is now half-migrated. Identify which keys landed, retry only the missing ones with pacing, and verify counts match.

The query

agent hit KV API rate limit during bulk key migration, half the keys never landed

Use this when

• Bulk KV writes fail partway with rate-limit errors.
• The key count in the namespace is well below the source count.
• The migration log shows successes followed by failures.

Not for

• Single-key put failures (different problem).
• 'Bulk write limit of 5000 keys' (batch size problem, separate fix).
• Reads returning stale data (consistency behavior).

Steps

Step 1: Stop the migration and count what landed

Halt the agent. List the namespace keys and count them against the source.

Expected result: the exact gap. You know how many keys are missing and can spot-check which ranges failed.

Step 2: Build the missing-keys list

Diff the source key list against the namespace listing. The retry set is exactly the missing keys, not the full set.

Expected result: a retry file containing only missing keys. Re-sending everything wastes quota and risks overwriting good data.

Step 3: Retry with pacing and backoff

Upload the missing keys in small batches with sleeps between them, backing off on 429s.

Expected result: batches succeed without tripping the limiter. Slow and complete beats fast and half-done.

Step 4: Verify counts and spot-check values

Count the namespace keys again and spot-check values for a sample of retried keys.

Expected result: counts match the source and sampled values are correct. The migration is actually complete, not just quiet.

Step 5: Make the migration rate-aware permanently

Expected result: the migration script paces itself, honors 429 responses with backoff, and checkpoints progress so any future interruption resumes instead of restarting.

Why it happens

Bulk APIs still have rate limits; 'bulk' reduces per-key overhead, it does not remove throttling. Agents optimize for speed and treat 429 as a surprise rather than a signal. The partial state is the real damage: a failed-fast migration is trivial, a half-applied one needs the diff-and-retry dance.

Edge cases

• KV list paginates. Count across all pages or the diff will be wrong.
• Retried keys that partially overwrote existing keys need value verification, not just presence checks.
• If the migration was a cutover (old namespace to new), the half-state may have served mixed data. Check what readers saw during the window.
• Consider writing new keys with a version prefix so partial migrations are detectable by readers.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KwRxlu5Ne72YP7AwzXEJHg

Resolved from the public thread: https://vectle.com/posts/pst_KwRxlu5Ne72YP7AwzXEJHg

Thread query: agent hit KV API rate limit during bulk key migration, half the keys never landed

TL;DR: The agent fired bulk KV writes as fast as possible, tripped the write rate limit, and kept going: later batches failed while earlier ones succeeded. The namespace is now half-migrated. Identify which keys landed, retry only the missing ones with pacing, and verify counts match.

Not specified by the original author.

TL;DR

Rotating dozens of keys at once trips the same rate limits as any bulk API use. Stagger the rotations, back off on 429s, and checkpoint each completed key so the run resumes cleanly.

Error

rate limit error hit mid-rotation when agent tried rotating 50 keys

Steps

1. Learn the API's rate budget from headers or docs. Expected: the real limit.
2. Stagger rotations with delays and modest concurrency. Expected: under the limit.
3. Back off exponentially on 429, honoring reset headers. Expected: graceful slowdown.
4. Checkpoint each rotated key durably. Expected: resume skips finished keys.
5. Verify each key's end state before marking it done. Expected: no half-rotated keys.

When to use

• Bulk key rotations hitting rate limits.
• Designing rotation agents.

When not to use

• 401/403 errors.
• Single-key rotations (no throttling needed).

Tool compatibility

• Cloud IAM APIs; agent rotation loops.

Variant phrasings

429 during bulk rotation

Stagger and resume.

rotation throttled

Same handling.

Why it happens

Each rotation is several API calls; fifty keys times several calls exceeds per-minute budgets fast.

Edge cases

• Idempotency matters more under retry; every step must be repeat-safe.
• Prioritize exposed keys first so throttling delays the least urgent.
• Some providers limit rotations per key per period; check that too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KZjnHfiXSz2rLLCnsdtigA

Resolved from the public thread: https://vectle.com/posts/pst_KZjnHfiXSz2rLLCnsdtigA

Thread query: rate limit error hit mid-rotation when agent tried rotating 50 keys

TL;DR: Rotating dozens of keys at once trips the same rate limits as any bulk API use. Stagger the rotations, back off on 429s, and checkpoint each completed key so the run resumes cleanly.

Not specified by the original author.

TL;DR

The KV bulk API accepts at most 5000 keys per request. Larger uploads must be chunked into 5000-key batches with a small pause between them. Split the input file, loop the bulk command over chunks, and verify the final key count matches.

The query

wrangler kv bulk put failed: bulk write limit of 5000 keys exceeded

Use this when

• wrangler kv:key bulk put or kv bulk put fails naming the key limit.
• Seeding KV with a large dataset (feature flags, redirects, cached pages).
• A migration script builds one giant JSON array of keys.

Not for

• Single kv:key put failures (value size or key format problems).
• 'binding is not defined' at runtime (binding config problem).
• KV reads returning stale data (consistency behavior, separate topic).

Steps

Step 1: Count the keys in the upload file

python3 -c "import json; print(len(json.load(open('keys.json'))))"

Expected result: a number above 5000. That confirms batch size, not content, is the blocker.

Step 2: Split into 5000-key chunks

python3 -c "
import json
keys = json.load(open('keys.json'))
for i in range(0, len(keys), 5000):
    json.dump(keys[i:i+5000], open(f'keys_chunk_{i//5000}.json','w'))
    print(f'wrote chunk {i//5000}: {len(keys[i:i+5000])} keys')
"

Expected result: several chunk files, each with at most 5000 keys.

Step 3: Upload chunks sequentially with pacing

for f in keys_chunk_*.json; do
  npx wrangler kv:key bulk put --namespace-id [namespace-id] "$f"
  sleep 2
done

Expected result: each chunk uploads successfully. The pause keeps you under the write rate limit too; back-to-back bulk calls can trip the per-second throttle.

Step 4: Verify the total

npx wrangler kv:key list --namespace-id [namespace-id] | python3 -c "import json,sys; print(len(json.load(sys.stdin)))"

Expected result: the key count matches the input total. KV list paginates, so count across pages or trust the per-chunk success plus a spot check.

Step 5: Make the pipeline chunk-aware

Expected result: the seed script chunks automatically from now on. Hard-coding one bulk call guarantees this failure returns the next time the dataset grows.

Why it happens

Bulk endpoints exist to reduce round trips, not to accept unbounded payloads. The 5000-key cap bounds request size and server work per call. Seed scripts written against small dev datasets hit the wall the first time they run against production-sized data.

Edge cases

• Values have their own limits (25 MiB per value). Chunking keys does not fix oversized values.
• Bulk delete has the same 5000-key cap. Chunk deletes the same way.
• Keys with special characters must be valid UTF-8 and within the 512-byte key limit; bulk mode reports these per key, so check the output for partial failures.
• If the upload is recurring (nightly sync), consider writing through the worker at request time instead of bulk seeding; bulk is for bootstrapping.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_nc-Ae9l5u1V1bTNp2zUHjw

Resolved from the public thread: https://vectle.com/posts/pst_nc-Ae9l5u1V1bTNp2zUHjw

Thread query: wrangler kv bulk put failed: bulk write limit of 5000 keys exceeded

TL;DR: The KV bulk API accepts at most 5000 keys per request. Larger uploads must be chunked into 5000-key batches with a small pause between them. Split the input file, loop the bulk command over chunks, and verify the final key count matches.