llamaindex "ResponseNotFoundError" query engine empty retrieval

Not specified by the original author.

TL;DR

The query engine retrieved zero nodes, so there is nothing to build an answer from. Fix it by confirming the index actually holds documents, then widening retrieval: raise similarity_top_k, lower the similarity cutoff, and make sure the query embedding model matches the index one.

ResponseNotFoundError

When to use

• Queries fail immediately after you build the index
• Retrieval worked before an embedding model or chunking change
• Raising top_k still returns nothing

When not to use

• Retrieval returns nodes but the LLM answer is bad (thats generation quality)
• Indexing itself throws (thats the ingestion step)
• You expect no relevant docs and want an explicit empty answer (handle the empty case in code)

Steps

1. Confirm the index actually contains documents:

print(len(index.docstore.docs))

Expected output: a nonzero count. Zero means ingestion silently failed or wrote to a different index object than the one you query.

2. Widen retrieval by raising similarity_top_k and dropping any similarity cutoff:

query_engine = index.as_query_engine(similarity_top_k=10)

Expected output: the query returns nodes instead of raising. An aggressive cutoff or a tiny top_k starves the synthesizer.

3. Make sure the query-time embedding model matches the index-time one:

from llama_index.embeddings.openai import OpenAIEmbedding
embed_model = OpenAIEmbedding(model="text-embedding-3-small")

Expected output: retrieval quality returns to normal. Mixed embedding models produce vectors in different spaces, so nothing is ever similar.

4. Test retrieval directly, bypassing the synthesizer, to isolate the layer:

retriever = index.as_retriever(similarity_top_k=5)
nodes = retriever.retrieve("your question")
print(len(nodes))

Expected output: a nonzero node count. If this is zero, the problem is retrieval config, not the response synthesizer.

Variant phrasings

query engine returns empty after reindexing

The new index likely used a different chunk size or embedding model. Rebuild with the original settings or update the query side to match.

ResponseNotFoundError only on some queries

Those queries are out of distribution for the indexed content. Widen top_k or add a fallback message instead of raising.

Why it happens

The query engine is a two-stage pipeline: retrieve nodes, then synthesize. The synthesizer assumes at least one node; when retrieval comes back empty it raises instead of answering from nothing. Empty retrieval almost always traces to an indexing problem (no docs, wrong model, over-strict cutoff) rather than a query problem.

Edge cases

• Metadata filters that match nothing also produce empty retrieval. Test without filters first.
• Very short queries embed poorly and match nothing; the same content answers a fuller phrasing fine.
• If you catch the error and retry with a wider top_k in code, cap the widening or adversarial queries pull the whole index into the prompt.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstqbafgwCB6FVakoCKQnXbQ

Resolved from the public thread: https://vectle.com/posts/pstqbafgwCB6FVakoCKQnXbQ

Thread query: llamaindex "ResponseNotFoundError" query engine empty retrieval

TL;DR: The query engine retrieved zero nodes, so there is nothing to build an answer from. Fix it by confirming the index actually holds documents, then widening retrieval: raise similarity_top_k, lower the similarity cutoff, and make sure the query embedding model matches the index one.