## TL;DR
The agent's upsert loop re-embeds every document on every pass, including ones that never changed. Add a dedupe guard: hash each document's content before embedding, compare against the hash stored alongside the vector, and skip the embed call when the hash matches.

## Symptom
An agent running a vector upsert loop re-embeds the same documents every cycle. Embedding spend climbs on every run, ingestion never converges, and unchanged documents get fresh vectors with new ids, which can disturb stable retrieval ordering.

## When to use
- Ingestion costs climb on every run with no new documents
- Unchanged docs get re-embedded needlessly
- The upsert loop never converges to a no-op pass

## When not to use
- Documents genuinely changed (those must be re-embedded)
- You migrated embedding models (all vectors need rebuilding, hashes do not help)
- The loop re-fetches but does not re-embed (thats a fetch-side problem)

## Steps

1. Hash each document's content before any embedding call:

```python
import hashlib
def content_hash(text):
    return hashlib.sha256(text.encode("utf-8")).hexdigest()
```
Expected output: identical documents always produce the identical hash. Hash the raw text, not the metadata, so metadata-only edits do not trigger re-embeds.

2. Store the hash alongside each vector at upsert time:

```python
point = {"id": doc_id, "vector": embedding, "payload": {"text": text, "hash": content_hash(text)}}
```
Expected output: every stored point carries the hash of the text it was embedded from. Without this, the guard has nothing to compare against.

3. Skip the embed call when the stored hash matches:

```python
stored = fetch_payload(doc_id)
if stored and stored.get("hash") == content_hash(text):
    continue  # unchanged, skip embedding
embedding = embed(text)
```
Expected output: steady-state runs embed zero documents. The loop converges to a cheap no-op pass instead of a full re-embed.

4. Backfill hashes for points written before the guard existed, or treat missing hashes as changed exactly once:

```python
if stored is None or "hash" not in stored:
    embedding = embed(text)  # one-time backfill
```
Expected output: one full pass re-embeds everything and stores hashes; every pass after that is incremental.

## Variant phrasings

### vector db upsert re-embedding everything nightly
Same fix. Nightly jobs without a change check re-embed the world; the hash guard makes them incremental.

### duplicate vectors accumulating in the collection
The loop upserts without stable ids, so each pass adds new points. Combine the hash guard with deterministic ids to stop the growth.

## Why it happens
Agents write upsert loops as fetch-all then embed-all because it is the simplest correct shape. Nothing in that shape distinguishes changed from unchanged documents, so every pass pays the full embedding cost. The missing piece is a cheap change signal, and a content hash is the cheapest reliable one.

## Edge cases
- Whitespace-only or encoding changes alter the hash. Normalize (strip, unify newlines) before hashing if those edits should not count as changes.
- Metadata-only updates (titles, tags) need a separate path: update the payload without re-embedding.
- Hash collisions are theoretically possible but sha256 makes them a non-issue in practice; do not downgrade to a short hash to save space.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_f0X6dCs-YUmlMnuGC0j5OA