ChromaDB ValueError: Cannot submit more than 41,666 embeddings at once

TL;DR

For "ValueError: Cannot submit more than 41,666 embeddings at once. Please submit your embeddings in batches of size 41,666 or less.": please submit your embeddings in batches of size 41,666 or less." Nothing else changed; the reporter was ingesting offline documents with langchain 0.0.274 and privateGPT-style code. The new limit was introduced because chromadb 0.4.x moved to SQLite, which limits the length of a single SQL query.

ValueError: Cannot submit more than 41,666 embeddings at once. Please submit your embeddings in batches of size 41,666 or less.

Steps

1. After upgrading from chromadb 0.3.26 to 0.4.7, bulk ingestion started failing with "ValueError: Cannot submit more than 41,666 embeddings at once. Please submit your embeddings in batches of size 41,666 or less." Nothing else changed; the reporter was ingesting offline documents with langchain 0.0.274 and privateGPT-style code. The new limit was introduced because chromadb 0.4.x moved to SQLite, which limits the length of a single SQL query.
2. This is intentional, not a bug: chromadb 0.4.x moved to SQLite, which limits the length of a single SQL query, so a hard batch cap of 41,666 embeddings was introduced around 0.4.6 (confirmed by a maintainer in the thread). Submit your embeddings in batches of 41,666 or fewer per add/upsert call; the error message states the exact cap.

When to use

You are seeing this: After upgrading from chromadb 0.3.26 to 0.4.7, bulk ingestion started failing with "ValueError: Cannot submit more than 41,666 embeddings at once. Use this skill when you run into "ChromaDB ValueError: Cannot submit more than 41,666 embeddings at once".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: chromadb 0.3.26, 0.4.7, langchain 0.0.274, around 0.4.6. If you are on something much newer or older, the details may have shifted.

Why this happens

The new limit was introduced because chromadb 0.4.x moved to SQLite, which limits the length of a single SQL query.

In the thread ChromaDB ValueError: Cannot submit more than 41,666 embeddings at once, the question was: After upgrading from chromadb 0.3.26 to 0.4.7, bulk ingestion started failing with "ValueError: Cannot submit more than 41,666 embeddings at once. Please submit your embeddings in batches of size 41,666 or less." Nothing else changed; the reporter was ingesting offline documents with langchain 0.0.274 and privateGPT-style code. The new limit was introduced because chromadb 0.4.x moved to SQLite, which limits the length of a single SQL query. The verified resolution: This is intentional, not a bug: chromadb 0.4.x moved to SQLite, which limits the length of a single SQL query, so a hard batch cap of 41,666 embeddings was introduced around 0.4.6 (confirmed by a maintainer in the thread). Submit your embeddings in batches of 41,666 or fewer per add/upsert call; the error message states the exact cap. Original discussion: https://vectle.com/threads/thr_h05k4xUh99EeKG1fCUxshw