Immutable solution version
ChromaDB DuplicateIDError on memgpt load (duplicate embedding IDs)
Complete solution
The load path was not deduplicating IDs before insert; the fix was merged in MemGPT PR #1001 and shipped in the nightly package (`pip install pymemgpt-nightly`) and the release that followed within days.
What was going on Running memgpt load with a large .jsonl dataset failed with "chromadb.errors.DuplicateIDError: Expected IDs to be unique, found duplicates of: ...". The dataset produced duplicate embedding IDs on load, and Chroma's validate_ids rejected the insert, leaving the data source unusable. Maintainer sarahwooders reproduced the same error on chromaDB 0.4.22 and confirmed the load path was not deduplicating IDs before insert.
What fixed it The load path was not deduplicating IDs before insert; the fix was merged in MemGPT PR #1001 and shipped in the nightly package (pip install pymemgpt-nightly) and the release that followed within days. Update to a version containing #1001. If you are stuck on an older version, deduplicate your .jsonl by ID before loading.
Evidence
In the thread, the question was: Running memgpt load with a large .jsonl dataset failed with "chromadb.errors.DuplicateIDError: Expected IDs to be unique, found duplicates of: ...". The dataset produced duplicate embedding IDs on load, and Chroma's validate_ids rejected the insert, leaving the data source unusable. Maintainer sarahwooders reproduced the same error on chromaDB 0.4.22 and confirmed the load path was not deduplicating IDs before insert. The verified resolution: The load path was not deduplicating IDs before insert; the fix was merged in MemGPT PR #1001 and shipped in the nightly package (pip install pymemgpt-nightly) and the release that followed within days. Update to a version containing #1001. If you are stuck on an older version, deduplicate your .jsonl by ID before loading. Thread: https://vectle.com/threads/thr_eNvB5pIZrfo30I6jsKSOtg
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_VmDsxZDpJW0P917IHKesvw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">ChromaDB DuplicateIDError on memgpt load (duplicate embedding IDs) <span>— Vectle</span></a>