Immutable solution version
Pinecone diagnosis: upsert 400s on large batches
Complete solution
Symptom: big upsert batches fail with 400 Invalid Argument while small ones work. The request exceeds the size cap; split into smaller batches.
Diagnosis
Symptom
Upserting thousands of vectors in one call returns 400; the same vectors succeed in smaller batches.
Likely cause
Single upsert requests are capped at about 2 MB. Large batches with rich metadata exceed it. The 400 is deterministic for the same payload.
Confirm it
- Halve the batch until it succeeds; the threshold you find is your effective cap for that vector shape.
- Estimate: dimension times 4 bytes per vector plus metadata; keep batches comfortably under 2 MB.
- Check per-record limits too: 512-char ids, 40 KB metadata, 2048 sparse non-zeros.
Fix
Batch by byte size, not just count: accumulate vectors until near the cap, then send. Parallelize across batches for throughput, with backoff on 429.
Verify the fix
Ingest the full dataset with the new batcher and assert describe_index_stats totals equal the source count.
Evidence
Pinecone's upsert guide documents the request size cap; the error-handling guide defines 400 and forbids retrying it. Limits reference: 512-char ids, 40 KB metadata. Verified against live docs 2026-09-26.
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Maintainer review
No maintainer verification is recorded for this version.
A maintainer review records the library version checked. It does not assert that it is the latest upstream release. Reported outcomes are separate evidence.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_DFgA2tbOsEqR4G5rmTceIw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Pinecone diagnosis: upsert 400s on large batches <span>— Vectle</span></a>