Pinecone workflow: bulk upsert millions of vectors with batching

Bulk upsert that finishes

The shape of the job

read source -> embed -> batch (by bytes, not just count) -> upsert in parallel -> verify counts

Steps

1. Embed first, or stream. For millions of vectors, embed in a separate pass and store the vectors, or stream embedding into the batcher. Do not re-embed on retry.
2. Batch by bytes. A request caps at about 2 MB. Accumulate vectors until the batch nears the cap (dimension * 4 bytes per float plus metadata), then send. Fixed counts of 100 or 1000 are fine only if your vectors are small; byte-batching is robust to metadata size.
3. Parallelize with a cap. 4 to 10 concurrent upserts is a sane starting point. More than that and 429s eat the gains.
4. Retry 429 and 5xx with backoff. Never retry 400: a 400 is a malformed batch, and resending it is a loop.
5. Checkpoint. Record the last successfully sent batch id. On crash, resume from the checkpoint; upsert is idempotent by id, so re-sending a batch is safe.
6. Verify. describe_index_stats() total must equal the source count. Sample-fetch random ids.

Traps

1. One vector per upsert call: works for 100 vectors, 429s and crawls at 100k.
2. Batching by count with heavy metadata: 1000 vectors times 30 KB metadata blows the 2 MB cap and every batch 400s.
3. No checkpointing: a crash at 90 percent restarts the whole ingest.
4. Embedding inside the retry loop: a transient upsert failure re-embeds, doubling embedding cost.

Pinecone's upsert guide documents the request size cap; the error-handling guide prescribes batching to reduce request counts and backoff for 429/5xx. Verified against live docs 2026-09-26.