**TL;DR:** Symptom: big upsert batches fail with 400 Invalid Argument while small ones work. The request exceeds the size cap; split into smaller batches. Confirm it 1. Check per-record limits too: 512-char ids, 40 KB metadata, 2048 sparse non-zeros. ## The fix # Diagnosis ## Symptom Upserting thousands of vectors in one call returns 400; the same vectors succeed in smaller batches. ## Likely cause Single upsert requests are capped at about 2 MB. Large batches with rich metadata exceed it. The 400 is deterministic for the same payload. ## Confirm it 1. Halve the batch until it succeeds; the threshold you find is your effective cap for that vector shape. 2. Estimate: dimension times 4 bytes per vector plus metadata; keep batches comfortably under 2 MB. 3. Check per-record limits too: 512-char ids, 40 KB metadata, 2048 sparse non-zeros. ## Fix Batch by byte size, not just count: accumulate vectors until near the cap, then send. Parallelize across batches for throughput, with backoff on 429. ## Verify the fix Ingest the full dataset with the new batcher and assert `describe_index_stats` totals equal the source count. ## When to use this - This covers exactly what the title says: Pinecone diagnosis. - You are setting this up for the first time, or auditing an existing setup. - You want the key gotchas in one place before you start. ## When not to use this - You are doing a different workflow with Pinecone; these steps are specific to the title above. - You need the full reference docs; this is the short path, not the manual. ## Compatibility - Not pinned to a specific version; follows current Pinecone behavior. --- ## Vectle citations - Exact version: - Canonical HTML skill page: - Current Markdown URL: ## Selected public attribution and original notices/references ```json { "author": "issueatlas", "client": null, "notices_and_references": [ "https://docs.pinecone.io/guides/index-data/upsert-data", "https://docs.pinecone.io/guides/production/error-handling" ] } ```