Weaviate BM25 returns no results in some situations (inverted index bug, fixed in #3592)

TL;DR

This was a real inverted-index bug that only surfaced in v1.21, tied to LSM flush/compaction behavior; it is explained and fixed in weaviate PR #3592.

Steps

1. BM25 search on a Weaviate class (French legal articles, text2vec-transformers vectorizer, stopwords preset "none") silently returned no results for queries that should have matched. The failure did not appear immediately after writes but surfaced later, after LSM flush/compaction cycles. The reporter confirmed the pattern after maintainer etiennedi diagnosed it, and noted the bug appeared after upgrading to v1.21.
2. This was a real inverted-index bug that only surfaced in v1.21, tied to LSM flush/compaction behavior; it is explained and fixed in weaviate PR #3592. If BM25 stops returning results some time after writes (not immediately), upgrade to a version containing the #3592 fix.

When to use

You are seeing this: The failure did not appear immediately after writes but surfaced later, after LSM flush/compaction cycles. Use this skill when you run into "Weaviate BM25 returns no results in some situations (inverted index bug, fixed in #3592)".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: v1.21. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Asked in the thread (https://vectle.com/threads/thrtDLLmW-51athMOkYSUeKEw): BM25 search on a Weaviate class (French legal articles, text2vec-transformers vectorizer, stopwords preset "none") silently returned no results for queries that should have matched. The failure did not appear immediately after writes but surfaced later, after LSM flush/compaction cycles. The reporter confirmed the pattern after maintainer etiennedi diagnosed it, and noted the bug appeared after upgrading to v1.21. The verified answer: This was a real inverted-index bug that only surfaced in v1.21, tied to LSM flush/compaction behavior; it is explained and fixed in weaviate PR #3592. If BM25 stops returning results some time after writes (not immediately), upgrade to a version containing the #3592 fix. Source: https://github.com/weaviate/weaviate/issues/3517 Thread: https://vectle.com/threads/thrtDLLmW-51athMOkYSUeKEw