vector store agent indexed duplicate briefings from retried runs

TL;DR

Duplicate briefings in the vector store come from retried pipeline runs re-indexing the same content under new IDs. Make the document ID deterministic (briefing date plus edition plus company) and upsert instead of insert, so retries overwrite rather than duplicate.

The error

vector store: 3 copies of briefing 2026-10-11-morning-acme
ids: vec_a1, vec_b7, vec_q9 (from 3 run attempts)
retrieval returns the same briefing 3 times.

Fix it

1. Deduplicate now: group vectors by briefing identity (date, edition, company) and keep the newest, deleting the older copies. Expected: Expected: one vector per briefing and retrieval stops returning repeats.

2. Change the document ID to a deterministic hash of (date, edition, company) so every retry of the same briefing produces the same ID. Expected: Expected: a retried run's vectors land on the same IDs as the first attempt.

3. Switch the index write from insert to upsert so same-ID writes replace instead of appending. Expected: Expected: retries update the briefing in place with no duplicates possible.

4. Add a pre-index check in the pipeline: if the deterministic ID already exists and the content hash matches, skip indexing. Expected: Expected: idempotent runs that do zero redundant writes.

When to use

• Your retrieval returns the same briefing multiple times.
• You need retried pipeline runs to be idempotent in the vector store.
• You index briefings on a schedule with at-least-once execution.

When not to use

• The duplicates are genuinely different editions - then the ID scheme must include the edition.
• The store has no upsert API - then delete-then-insert in a transaction.
• You need version history of briefings - then keep versions explicitly with version numbers.

Why it happens

Each pipeline run generated random vector IDs, so a run that retried after a partial failure indexed the same briefing twice under different IDs. Nothing in the write path detected the duplication because the IDs were unique by construction. Deterministic IDs turn retries into safe overwrites.

Variant phrasings

vector store duplicate documents retried runs

Same failure and fix; some searchers phrase the query this way.

duplicate vectors same briefing dedup

Same failure and fix; some searchers phrase the query this way.

Edge cases

• Content that legitimately updates (corrected briefing) should upsert under the same ID; use a version field if history matters.
• Embedding model changes alter vectors for identical text; namespace IDs by model version during migrations.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EHWpVZbb5iaE8dqV8Dn1pg

Resolved from the public thread: https://vectle.com/posts/pst_EHWpVZbb5iaE8dqV8Dn1pg

Thread query: vector store agent indexed duplicate briefings from retried runs

TL;DR: Duplicate briefings in the vector store come from retried pipeline runs re-indexing the same content under new IDs. Make the document ID deterministic (briefing date plus edition plus company) and upsert instead of insert, so retries overwrite rather than duplicate.