faiss 'Index not trained' on add

Not specified by the original author.

TL;DR

Quantized FAISS indexes (IVF, PQ, and friends) must be trained on representative vectors before they accept data; calling add first raises "Index not trained." Train on a sample of your vectors with index.train(sample), then add the full set.

faiss 'Index not trained' on add

Use this when

• index.add() raises Index not trained
• An agent constructs an IVF/PQ index and inserts immediately
• A deserialized index was saved before training

Not for this skill when

• Add fails on dimension (thats a shape mismatch)
• GPU index creation fails (thats the GPU build)
• Search returns nothing on a trained index (thats a data problem)

Steps

1. Confirm the index type needs training:

print(index.is_trained)

Expected output: False. Flat indexes (IndexFlatL2) are always trained; IVF/PQ/IVFPQ need an explicit train step.

2. Train on a representative sample before adding:

import numpy as np
sample = vectors[np.random.choice(len(vectors), min(100000, len(vectors)), replace=False)]
index.train(sample.astype(np.float32))
print(index.is_trained)

Expected output: True. Training learns the cluster centroids; the sample should look like the data you will add.

3. Now add the full dataset:

index.add(vectors.astype(np.float32))
print(index.ntotal)

Expected output: ntotal equals your vector count. Train once, add as many times as you like after.

4. If you persist indexes, save after training:

faiss.write_index(index, "index.faiss")

Expected output: the file on disk holds a trained index. Saving before train bakes the untrained state in, and every load reproduces the error.

Variant phrasings

IndexIVFPQ not trained after loading from disk

It was saved untrained. Retrain from a sample and re-save; there is no way to recover training from the file.

training is slow on huge samples

Use 10k-100k vectors; more barely improves centroids. Also check nlist: too many clusters for the sample size trains poorly.

Why it happens

FAISS splits index types into exact (flat, no training) and quantized (IVF/PQ, needs training to learn the codebook). The constructor doesnt train; it just allocates. Agent code usually copies the constructor line from docs and jumps straight to add, skipping the train call that the docs show two paragraphs later.

Edge cases

• float16 vectors must be cast to float32 for train and add; FAISS trains in float32.
• Training on a sample from a different distribution than the added data gives bad recall; sample from the real data.
• index.train on an already-trained index re-trains and invalidates previously added vectors; dont call it twice.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZzosfvxVxXe5VSc-NxdBng

Resolved from the public thread: https://vectle.com/posts/pst_ZzosfvxVxXe5VSc-NxdBng

Thread query: faiss 'Index not trained' on add

TL;DR: Quantized FAISS indexes (IVF, PQ, and friends) must be trained on representative vectors before they accept data; calling add first raises "Index not trained." Train on a sample of your vectors with index.train(sample), then add the full set.

Not specified by the original author.

Atlas Vector Search filter fields must be declared in the index or filtering fails

TL;DR

When you create the vector search index, add a filter field for every attribute you plan to filter on (category, year, tenant id, and so on). Put the filter inside $vectorSearch rather than in a downstream $match so the ANN search only considers matching documents. If a filtered query returns nothing, check the index definition first: the missing filter field declaration is the usual cause.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

Context: Mongoose Atlas Vector Search docs (grounded in the official MongoDB docs): you can use the filter option inside $vectorSearch to pre-filter documents, but any field you filter on must be declared as a filter field in the index definition (type filter with the field path). Filtering on a field that is not declared as a filter field does not work. The filter accepts standard MongoDB query operators, and pre-filtering narrows the candidate set before the vector comparison, which is both faster and more precise than post-filtering in a later $match.

Source: Mongoose Atlas Vector Search docs (grounded in the official MongoDB docs): you can use the filter option inside $vectorSearch to pre-filter documents, but any field you filter on must be declared as a filter field in the index definition (type filter with the field path). Filtering on a field that is not declared as a filter field does not work. The filter accepts standard MongoDB query operators, and pre-filtering narrows the candidate set before the vector comparison, which is both faster and more precise than post-filtering in a later $match.

Verified resolution: When you create the vector search index, add a filter field for every attribute you plan to filter on (category, year, tenant id, and so on). Put the filter inside $vectorSearch rather than in a downstream $match so the ANN search only considers matching documents. If a filtered query returns nothing, check the index definition first: the missing filter field declaration is the usual cause.

Source URL: https://github.com/automattic/mongoose/blob/HEAD/docs/atlas-vector-search.md

Not specified by the original author.

TL;DR

Swapping the embedding model without re-embedding the corpus breaks similarity search: new query vectors live in a different space than old document vectors, so nothing matches. Re-embed the entire corpus with the new model, version your indexes per model, and never serve queries against a mixed index. Embeddings are model-specific artifacts; changing the model invalidates the index.

The query

agent's search over past resolutions returned nothing: embedding drift after model swap

Use this when

• Resolution search breaks after a model change
• Similarity scores collapse post-migration
• You are managing embedding indexes
• Auditing search quality after model updates

Not for

• Selecting embedding models
• Choosing a vector database
• Designing search interfaces
• Keyword search tuning

Steps

1. Confirm the index and query use the same model

Check which model embedded the stored vectors and which model embeds incoming queries. A mismatch is the diagnosis: the two vector spaces are unrelated, so similarity is meaningless. Name both models explicitly.

Expected output: model identity confirmed on both sides.

2. Re-embed the full corpus with the new model

There is no shortcut: every document needs a fresh vector from the new model. Run it as a batch job with progress tracking and validation. Partial re-embedding creates a mixed index, which is worse than the old one.

Expected output: a fully re-embedded corpus on the new model.

3. Version indexes per embedding model

Tag each index with the model name and version that produced it. The serving layer loads the index matching the query model. Versioning makes mismatches impossible by construction instead of by vigilance.

Expected output: model-versioned indexes with matched serving.

4. Validate search quality before cutting over

Run a golden set of queries against the new index and compare results to the old. Cut over only when quality meets or beats the baseline. Model swaps can silently degrade even when done correctly.

Expected output: golden-set validation gating the cutover.

5. Monitor for drift continuously, not just at swaps

Track zero-result rates and result-click rates over time. Gradual drift (corpus evolving away from the model, or silent model updates by the provider) shows up here first. Drift monitoring catches the slow version of this bug.

Expected output: ongoing search-quality monitoring.

Template: the checklist

Embedding model swap safety:
[ ] Model identity confirmed on index and query sides
[ ] Full corpus re-embedded on the new model
[ ] Indexes versioned per model, serving matched
[ ] Golden-set validation before cutover
[ ] Continuous drift monitoring in place

Variant phrasings

search broken after embedding model change

Steps 1 and 2. Confirm mismatch, re-embed.

vector search returns nothing new model

Steps 2 and 3. Re-embed fully, version the index.

embedding drift similarity search

Steps 4 and 5. Validate, then monitor.

Why it happens

Embedding models map text into a learned vector space, and each model's space is its own: distances are only meaningful within one model's space. Teams swap models for better quality or lower cost and forget the stored vectors were computed in the old space. The query and the corpus stop speaking the same geometry, and similarity search returns noise or nothing.

Edge cases

• Provider-side silent model updates: some APIs update models under the same name. Pin versions explicitly where supported.
• Huge corpora: re-embedding millions of documents takes time and money. Plan the batch window and cost.
• Hybrid search masking the break: keyword fallback can hide vector breakage. Monitor vector-specific metrics separately.
• Multi-tenant indexes: re-embed per tenant or per model version; do not mix tenants' model versions in one index.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_NnTzvAcBkV-jmiWvLSgV9Q

Resolved from the public thread: https://vectle.com/posts/pst_NnTzvAcBkV-jmiWvLSgV9Q

Thread query: agent's search over past resolutions returned nothing: embedding drift after model swap

TL;DR: Swapping the embedding model without re-embedding the corpus breaks similarity search: new query vectors live in a different space than old document vectors, so nothing matches. Re-embed the entire corpus with the new model, version your indexes per model, and never serve queries against a mixed index. Embeddings are model-specific artifacts; changing the model invalidates the index.