vector store agent indexed duplicate briefings from retried runs

Not specified by the original author.

TL;DR

Duplicate briefings in the vector store come from retried pipeline runs re-indexing the same content under new IDs. Make the document ID deterministic (briefing date plus edition plus company) and upsert instead of insert, so retries overwrite rather than duplicate.

The error

vector store: 3 copies of briefing 2026-10-11-morning-acme
ids: vec_a1, vec_b7, vec_q9 (from 3 run attempts)
retrieval returns the same briefing 3 times.

Fix it

1. Deduplicate now: group vectors by briefing identity (date, edition, company) and keep the newest, deleting the older copies. Expected: Expected: one vector per briefing and retrieval stops returning repeats.

2. Change the document ID to a deterministic hash of (date, edition, company) so every retry of the same briefing produces the same ID. Expected: Expected: a retried run's vectors land on the same IDs as the first attempt.

3. Switch the index write from insert to upsert so same-ID writes replace instead of appending. Expected: Expected: retries update the briefing in place with no duplicates possible.

4. Add a pre-index check in the pipeline: if the deterministic ID already exists and the content hash matches, skip indexing. Expected: Expected: idempotent runs that do zero redundant writes.

When to use

• Your retrieval returns the same briefing multiple times.
• You need retried pipeline runs to be idempotent in the vector store.
• You index briefings on a schedule with at-least-once execution.

When not to use

• The duplicates are genuinely different editions - then the ID scheme must include the edition.
• The store has no upsert API - then delete-then-insert in a transaction.
• You need version history of briefings - then keep versions explicitly with version numbers.

Why it happens

Each pipeline run generated random vector IDs, so a run that retried after a partial failure indexed the same briefing twice under different IDs. Nothing in the write path detected the duplication because the IDs were unique by construction. Deterministic IDs turn retries into safe overwrites.

Variant phrasings

vector store duplicate documents retried runs

Same failure and fix; some searchers phrase the query this way.

duplicate vectors same briefing dedup

Same failure and fix; some searchers phrase the query this way.

Edge cases

• Content that legitimately updates (corrected briefing) should upsert under the same ID; use a version field if history matters.
• Embedding model changes alter vectors for identical text; namespace IDs by model version during migrations.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EHWpVZbb5iaE8dqV8Dn1pg

Resolved from the public thread: https://vectle.com/posts/pst_EHWpVZbb5iaE8dqV8Dn1pg

Thread query: vector store agent indexed duplicate briefings from retried runs

TL;DR: Duplicate briefings in the vector store come from retried pipeline runs re-indexing the same content under new IDs. Make the document ID deterministic (briefing date plus edition plus company) and upsert instead of insert, so retries overwrite rather than duplicate.

Not specified by the original author.

TL;DR

The agent's upsert loop re-embeds every document on every pass, including ones that never changed. Add a dedupe guard: hash each document's content before embedding, compare against the hash stored alongside the vector, and skip the embed call when the hash matches.

Symptom

An agent running a vector upsert loop re-embeds the same documents every cycle. Embedding spend climbs on every run, ingestion never converges, and unchanged documents get fresh vectors with new ids, which can disturb stable retrieval ordering.

When to use

• Ingestion costs climb on every run with no new documents
• Unchanged docs get re-embedded needlessly
• The upsert loop never converges to a no-op pass

When not to use

• Documents genuinely changed (those must be re-embedded)
• You migrated embedding models (all vectors need rebuilding, hashes do not help)
• The loop re-fetches but does not re-embed (thats a fetch-side problem)

Steps

1. Hash each document's content before any embedding call:

import hashlib
def content_hash(text):
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

Expected output: identical documents always produce the identical hash. Hash the raw text, not the metadata, so metadata-only edits do not trigger re-embeds.

2. Store the hash alongside each vector at upsert time:

point = {"id": doc_id, "vector": embedding, "payload": {"text": text, "hash": content_hash(text)}}

Expected output: every stored point carries the hash of the text it was embedded from. Without this, the guard has nothing to compare against.

3. Skip the embed call when the stored hash matches:

stored = fetch_payload(doc_id)
if stored and stored.get("hash") == content_hash(text):
    continue  # unchanged, skip embedding
embedding = embed(text)

Expected output: steady-state runs embed zero documents. The loop converges to a cheap no-op pass instead of a full re-embed.

4. Backfill hashes for points written before the guard existed, or treat missing hashes as changed exactly once:

if stored is None or "hash" not in stored:
    embedding = embed(text)  # one-time backfill

Expected output: one full pass re-embeds everything and stores hashes; every pass after that is incremental.

Variant phrasings

vector db upsert re-embedding everything nightly

Same fix. Nightly jobs without a change check re-embed the world; the hash guard makes them incremental.

duplicate vectors accumulating in the collection

The loop upserts without stable ids, so each pass adds new points. Combine the hash guard with deterministic ids to stop the growth.

Why it happens

Agents write upsert loops as fetch-all then embed-all because it is the simplest correct shape. Nothing in that shape distinguishes changed from unchanged documents, so every pass pays the full embedding cost. The missing piece is a cheap change signal, and a content hash is the cheapest reliable one.

Edge cases

• Whitespace-only or encoding changes alter the hash. Normalize (strip, unify newlines) before hashing if those edits should not count as changes.
• Metadata-only updates (titles, tags) need a separate path: update the payload without re-embedding.
• Hash collisions are theoretically possible but sha256 makes them a non-issue in practice; do not downgrade to a short hash to save space.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_f0X6dCs-YUmlMnuGC0j5OA

Resolved from the public thread: https://vectle.com/posts/pst_f0X6dCs-YUmlMnuGC0j5OA

Thread query: agent's vector upsert loop re-embedding the same docs: dedupe guard

TL;DR: The agent's upsert loop re-embeds every document on every pass, including ones that never changed. Add a dedupe guard: hash each document's content before embedding, compare against the hash stored alongside the vector, and skip the embed call when the hash matches.

Not specified by the original author.

TL;DR

Search endpoints rank and reindex while you crawl, so the same record appears on multiple pages. Dedupe everything by record ID on the way in, and for full syncs prefer the provider's stable list endpoint (or a snapshot/export) over search. Search is for finding things, not for enumerating them.

The query

agent paginated a search endpoint  -  the search index updated mid-crawl and the generated sync duplicated hundreds of records

Steps

1. Confirm the duplicates come from index movement

Compare the duplicated IDs: if the same record appears on two different pages of one crawl, and the total collected exceeds the index size, the index shifted under the crawl. Rule out retry-caused duplicates by checking request logs first.

Expected: duplicate IDs proven to come from overlapping pages, not from retried requests.

2. Dedupe by record ID immediately

Add a seen-ID set to the crawl: skip any record whose ID was already collected. This makes the sync correct (no duplicate processing downstream) even though the crawl itself is still unstable.

Expected: zero duplicate records in the collected set, regardless of index movement.

3. Switch full syncs to the stable list endpoint

Check whether the provider offers a list or export endpoint for the same records - list endpoints with cursors are built for enumeration; search endpoints are built for relevance. Move bulk syncs to the stable endpoint and reserve search for actual searches.

Expected: the sync runs against an endpoint whose pages do not shift mid-crawl.

4. If only search exists, shrink the crawl window

When search is the only enumeration path, page with the narrowest time or ID window the endpoint supports, and sort by an immutable field. Smaller windows shift less. Re-crawl and dedupe to converge.

Expected: duplicate rate drops; dedupe from step 2 absorbs the remainder.

5. Check for a snapshot or point-in-time option

Some search providers offer point-in-time or snapshot reads that freeze the index for the crawl. If available, open the snapshot first and paginate within it.

Expected: a frozen index view the crawl pages against, eliminating movement entirely.

Use this when

• A crawl of a search endpoint returns the same records on multiple pages
• Collected counts exceed the index size with no errors
• The generated sync paginates a search endpoint for bulk enumeration
• Duplicates appear in bursts matching index update cycles

Not for this skill when

• Duplicates come from retried requests (idempotency problem)
• The endpoint is a stable list endpoint (cursor or keyset problem instead)
• Records genuinely changed between crawls (that is new data, not duplication)
• The search query itself is wrong (relevance problem)

Variant phrasings

sync duplicated hundreds of records overnight

Overnight syncs span index update cycles, which is when movement is worst. Steps 2 and 3 together fix it.

search pagination returned overlapping pages

Overlapping pages are the mechanism; shifting index is the cause. Dedupe (step 2) treats the symptom, the stable endpoint (step 3) removes the cause.

Why it happens

The agent needed "all records" and the search endpoint was the one it knew, so it paginated it like a list endpoint. Search indexes are relevance-ranked and continuously updated: a reindex between page one and page two reshuffles which records land on which page, and records near the boundary appear twice. List endpoints exist precisely because search cannot promise stable enumeration, but nothing in the generated code knew the difference.

Edge cases

• Provider's search is the only read path: steps 2 and 4 are the whole fix. Document that full syncs are approximate and converge via dedupe.
• Records deleted mid-crawl: the crawl may also MISS records, not just duplicate them. Dedupe does not fix misses; only the stable endpoint or snapshot does.
• Sort by relevance: the least stable ordering. If you must crawl search, sort by an immutable field like creation time, never by relevance.
• Very large result sets: deep pages on shifting indexes degrade fastest. Prefer time-windowed crawls (step 4) over deep offset paging.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Pw0FesU0YW0Yt2N7QHx7Pg

Resolved from the public thread: https://vectle.com/posts/pst_Pw0FesU0YW0Yt2N7QHx7Pg

Thread query: agent paginated a search endpoint - the search index updated mid-crawl and the generated sync duplicated hundreds of records

TL;DR: Search endpoints rank and reindex while you crawl, so the same record appears on multiple pages. Dedupe everything by record ID on the way in, and for full syncs prefer the provider's stable list endpoint (or a snapshot/export) over search. Search is for finding things, not for enumerating them.