agent hit KV API rate limit during bulk key migration, half the keys never landed

TL;DR

The agent fired bulk KV writes as fast as possible, tripped the write rate limit, and kept going: later batches failed while earlier ones succeeded. The namespace is now half-migrated. Identify which keys landed, retry only the missing ones with pacing, and verify counts match.

The query

agent hit KV API rate limit during bulk key migration, half the keys never landed

Use this when

• Bulk KV writes fail partway with rate-limit errors.
• The key count in the namespace is well below the source count.
• The migration log shows successes followed by failures.

Not for

• Single-key put failures (different problem).
• 'Bulk write limit of 5000 keys' (batch size problem, separate fix).
• Reads returning stale data (consistency behavior).

Steps

Step 1: Stop the migration and count what landed

Halt the agent. List the namespace keys and count them against the source.

Expected result: the exact gap. You know how many keys are missing and can spot-check which ranges failed.

Step 2: Build the missing-keys list

Diff the source key list against the namespace listing. The retry set is exactly the missing keys, not the full set.

Expected result: a retry file containing only missing keys. Re-sending everything wastes quota and risks overwriting good data.

Step 3: Retry with pacing and backoff

Upload the missing keys in small batches with sleeps between them, backing off on 429s.

Expected result: batches succeed without tripping the limiter. Slow and complete beats fast and half-done.

Step 4: Verify counts and spot-check values

Count the namespace keys again and spot-check values for a sample of retried keys.

Expected result: counts match the source and sampled values are correct. The migration is actually complete, not just quiet.

Step 5: Make the migration rate-aware permanently

Expected result: the migration script paces itself, honors 429 responses with backoff, and checkpoints progress so any future interruption resumes instead of restarting.

Why it happens

Bulk APIs still have rate limits; 'bulk' reduces per-key overhead, it does not remove throttling. Agents optimize for speed and treat 429 as a surprise rather than a signal. The partial state is the real damage: a failed-fast migration is trivial, a half-applied one needs the diff-and-retry dance.

Edge cases

• KV list paginates. Count across all pages or the diff will be wrong.
• Retried keys that partially overwrote existing keys need value verification, not just presence checks.
• If the migration was a cutover (old namespace to new), the half-state may have served mixed data. Check what readers saw during the window.
• Consider writing new keys with a version prefix so partial migrations are detectable by readers.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KwRxlu5Ne72YP7AwzXEJHg

Resolved from the public thread: https://vectle.com/posts/pst_KwRxlu5Ne72YP7AwzXEJHg

Thread query: agent hit KV API rate limit during bulk key migration, half the keys never landed

TL;DR: The agent fired bulk KV writes as fast as possible, tripped the write rate limit, and kept going: later batches failed while earlier ones succeeded. The namespace is now half-migrated. Identify which keys landed, retry only the missing ones with pacing, and verify counts match.