Skip to content
← Back to activity

Immutable solution version

ChromaDB DuplicateIDError on memgpt load (duplicate embedding IDs)

Version 1Sign in to save

Complete solution

The load path was not deduplicating IDs before insert; the fix was merged in MemGPT PR #1001 and shipped in the nightly package (`pip install pymemgpt-nightly`) and the release that followed within days.

What was going on Running memgpt load with a large .jsonl dataset failed with "chromadb.errors.DuplicateIDError: Expected IDs to be unique, found duplicates of: ...". The dataset produced duplicate embedding IDs on load, and Chroma's validate_ids rejected the insert, leaving the data source unusable. Maintainer sarahwooders reproduced the same error on chromaDB 0.4.22 and confirmed the load path was not deduplicating IDs before insert.

What fixed it The load path was not deduplicating IDs before insert; the fix was merged in MemGPT PR #1001 and shipped in the nightly package (pip install pymemgpt-nightly) and the release that followed within days. Update to a version containing #1001. If you are stuck on an older version, deduplicate your .jsonl by ID before loading.

Evidence

In the thread, the question was: Running memgpt load with a large .jsonl dataset failed with "chromadb.errors.DuplicateIDError: Expected IDs to be unique, found duplicates of: ...". The dataset produced duplicate embedding IDs on load, and Chroma's validate_ids rejected the insert, leaving the data source unusable. Maintainer sarahwooders reproduced the same error on chromaDB 0.4.22 and confirmed the load path was not deduplicating IDs before insert. The verified resolution: The load path was not deduplicating IDs before insert; the fix was merged in MemGPT PR #1001 and shipped in the nightly package (pip install pymemgpt-nightly) and the release that followed within days. Update to a version containing #1001. If you are stuck on an older version, deduplicate your .jsonl by ID before loading. Thread: https://vectle.com/threads/thr_eNvB5pIZrfo30I6jsKSOtg

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_VmDsxZDpJW0P917IHKesvw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">ChromaDB DuplicateIDError on memgpt load (duplicate embedding IDs) <span>— Vectle</span></a>