## TL;DR

The agent fired bulk KV writes as fast as possible, tripped the write rate limit, and kept going: later batches failed while earlier ones succeeded. The namespace is now half-migrated. Identify which keys landed, retry only the missing ones with pacing, and verify counts match.

## The query

```text
agent hit KV API rate limit during bulk key migration, half the keys never landed
```

## Use this when

- Bulk KV writes fail partway with rate-limit errors.
- The key count in the namespace is well below the source count.
- The migration log shows successes followed by failures.

## Not for

- Single-key put failures (different problem).
- 'Bulk write limit of 5000 keys' (batch size problem, separate fix).
- Reads returning stale data (consistency behavior).

## Steps

### Step 1: Stop the migration and count what landed

Halt the agent. List the namespace keys and count them against the source.

Expected result: the exact gap. You know how many keys are missing and can spot-check which ranges failed.

### Step 2: Build the missing-keys list

Diff the source key list against the namespace listing. The retry set is exactly the missing keys, not the full set.

Expected result: a retry file containing only missing keys. Re-sending everything wastes quota and risks overwriting good data.

### Step 3: Retry with pacing and backoff

Upload the missing keys in small batches with sleeps between them, backing off on 429s.

Expected result: batches succeed without tripping the limiter. Slow and complete beats fast and half-done.

### Step 4: Verify counts and spot-check values

Count the namespace keys again and spot-check values for a sample of retried keys.

Expected result: counts match the source and sampled values are correct. The migration is actually complete, not just quiet.

### Step 5: Make the migration rate-aware permanently

Expected result: the migration script paces itself, honors 429 responses with backoff, and checkpoints progress so any future interruption resumes instead of restarting.

## Why it happens

Bulk APIs still have rate limits; 'bulk' reduces per-key overhead, it does not remove throttling. Agents optimize for speed and treat 429 as a surprise rather than a signal. The partial state is the real damage: a failed-fast migration is trivial, a half-applied one needs the diff-and-retry dance.

## Edge cases

- KV list paginates. Count across all pages or the diff will be wrong.
- Retried keys that partially overwrote existing keys need value verification, not just presence checks.
- If the migration was a cutover (old namespace to new), the half-state may have served mixed data. Check what readers saw during the window.
- Consider writing new keys with a version prefix so partial migrations are detectable by readers.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KwRxlu5Ne72YP7AwzXEJHg