ChromaDB DuplicateIDError on memgpt load (duplicate embedding IDs)

TL;DR

For "chromadb.errors.DuplicateIDError: Expected IDs to be unique, found duplicates of: ...": what was going on Running memgpt load with a large.jsonl dataset failed with "chromadb.errors.DuplicateIDError: Expected IDs to be unique, found duplicates of:...". The dataset produced duplicate embedding IDs on load, and Chroma's validate\_ids rejected the insert, leaving the data source unusable.

chromadb.errors.DuplicateIDError: Expected IDs to be unique, found duplicates of: ...

Steps

1. What was going on

Running memgpt load with a large .jsonl dataset failed with "chromadb.errors.DuplicateIDError: Expected IDs to be unique, found duplicates of: ...". The dataset produced duplicate embedding IDs on load, and Chroma's validate_ids rejected the insert, leaving the data source unusable. Maintainer sarahwooders reproduced the same error on chromaDB 0.4.22 and confirmed the load path was not deduplicating IDs before insert.

2. What fixed it

The load path was not deduplicating IDs before insert; the fix was merged in MemGPT PR #1001 and shipped in the nightly package (pip install pymemgpt-nightly) and the release that followed within days. Update to a version containing #1001. If you are stuck on an older version, deduplicate your .jsonl by ID before loading.

When to use

You are seeing this: Maintainer sarahwooders reproduced the same error on chromaDB 0.4.22 and confirmed the load path was not deduplicating IDs before insert. Use this skill when you run into "ChromaDB DuplicateIDError on memgpt load (duplicate embedding IDs)".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: chromaDB 0.4.22. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

In the thread, the question was: Running memgpt load with a large .jsonl dataset failed with "chromadb.errors.DuplicateIDError: Expected IDs to be unique, found duplicates of: ...". The dataset produced duplicate embedding IDs on load, and Chroma's validate_ids rejected the insert, leaving the data source unusable. Maintainer sarahwooders reproduced the same error on chromaDB 0.4.22 and confirmed the load path was not deduplicating IDs before insert. The verified resolution: The load path was not deduplicating IDs before insert; the fix was merged in MemGPT PR #1001 and shipped in the nightly package (pip install pymemgpt-nightly) and the release that followed within days. Update to a version containing #1001. If you are stuck on an older version, deduplicate your .jsonl by ID before loading. Thread: https://vectle.com/threads/thr_eNvB5pIZrfo30I6jsKSOtg