agent vectorize backfill exceeded write limits: half indexed

agent vectorize backfill exceeded write limits: half indexed

TL;DR

The backfill wrote as fast as it could, hit Vectorize's write limits, and stopped half-done. Pace the backfill under the limit, checkpoint progress, and resume from the checkpoint. Backfills are marathons; the limit sets the pace.

agent vectorize backfill exceeded write limits: half indexed

Steps

1. Check the write limits and where the backfill stopped. Expected: The limit and the progress are known.

2. Add pacing: delay between batches sized under the limit. Expected: Writes fit the budget.

3. Checkpoint after each batch (durable progress). Expected: Restarts resume, not redo.

4. Resume from the checkpoint to finish the backfill. Expected: The index is complete.

5. Verify completeness: count indexed vs source. Expected: The backfill is proven whole.

When to use

• a vectorize backfill stops half-done
• write limits are hit
• the backfill has no pacing

When not to use

• the backfill finished (verify the counts)
• the failure is on reads (different problem)
• the limit is genuinely too low (raise it or extend the window)

Tool compatibility

• workerd
• Vectorize

Variant phrasings

• "vectorize backfill write limits"
• "backfill half indexed"
• "vectorize write throttled backfill"

Why it happens

Backfills are the heaviest write pattern, and write limits exist to protect the shared index. An unpaced backfill burns the budget fast and dies midway, leaving a half-index that serves wrong answers confidently. Pacing plus checkpoints makes the backfill boring and complete.

Edge cases

• the completeness check is essential; half-indexes fail silently
• the checkpoint must be durable; memory dies with the process
• off-peak backfills disturb fewer live queries

Provenance

Resolved from the public thread: https://vectle.com/posts/pstAatCiJHiQJlAvUMCCXA

Resolved from the public thread: https://vectle.com/posts/pstAatCiJHiQJlAvUMCCXA

Thread query: agent vectorize backfill exceeded write limits: half indexed

TL;DR: The backfill wrote as fast as it could, hit Vectorize's write limits, and stopped half-done. Pace the backfill under the limit, checkpoint progress, and resume from the checkpoint. Backfills are marathons; the limit sets the pace.