Immutable solution version
HNSW index broken in v3 beta: rebuild chain plus brute-force fallback
Complete solution
How to hNSW index broken in v3 beta: rebuild chain plus brute-force fallback. Verified in gh:surrealdb/surrealdb#6872.
What this teaches
How to hNSW index broken in v3 beta: rebuild chain plus brute-force fallback.
The problem
Issue surrealdb/surrealdb#6872 (closed, 35 comments): HNSW index creation (DEFINE INDEX ... HNSW) fails with an IO error after the underlying surrealkv data files become corrupted or go missing. The error is unrecoverable: REMOVE INDEX succeeds but the subsequent DEFINE INDEX still fails. Regular queries work fine, as do FULLTEXT indexes. Only HNSW index operations are affected. Error: 'There was a problem with the key-value store: ... IO error: No such file or directory (os error 2)'.
What worked (verified answer)
SurrealDB v3 beta: HNSW index breaks with IO errors ('There was a problem with the key-value store ... No such file or directory') after the underlying data files corrupt or go missing. REMOVE INDEX succeeds but re-defining the index still fails, and REBUILD INDEX is broken too. No official root cause was found. Reporter's workaround chain that got HNSW usable: (1) data inserted after DEFINE INDEX ... HNSW is invisible to KNN, so auto-rebuild the index every 25 new embeddings via REMOVE INDEX + DEFINE INDEX; (2) use the two-parameter KNN form [|k,ef|] instead of the single-param <|k|> form, which is broken; (3) in your search function, try HNSW KNN first and fall back to brute-force vector distance when the index looks stale or corrupt.
Source
gh:surrealdb/surrealdb#6872 - https://github.com/surrealdb/surrealdb/issues/6872
Evidence
In gh:surrealdb/surrealdb#6872, the reporter ran into: Issue surrealdb/surrealdb#6872 (closed, 35 comments): HNSW index creation (DEFINE INDEX ... HNSW) fails with an IO error after the underlying surrealkv data files become corrupted or go missing. The error is unrecoverable: REMOVE INDEX succeeds but the subsequent DEFINE INDEX still fails. Regular queries work fine, as do FULLTEXT indexes. Only HNSW index operations are affected. Error: 'There was a problem with the key-value store: ... IO error: No such file or directory (os error 2)'. The verified answer was: SurrealDB v3 beta: HNSW index breaks with IO errors ('There was a problem with the key-value store ... No such file or directory') after the underlying data files corrupt or go missing. REMOVE INDEX succeeds but re-defining the index still fails, and REBUILD INDEX is broken too. No official root cause was found. Reporter's workaround chain that got HNSW usable: (1) data inserted after DEFINE INDEX ... HNSW is invisible to KNN, so auto-rebuild the index every 25 new embeddings via REMOVE INDEX + DEFINE INDEX; (2) use the two-parameter KNN form [|k,ef|] instead of the single-param <|k|> form, which is broken; (3) in your search function, try HNSW KNN first and fall back to brute-force vector distance when the index looks stale or corrupt. Source issue: https://github.com/surrealdb/surrealdb/issues/6872
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_6gF-QQ57r1XdscBDCpqj9Q" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">HNSW index broken in v3 beta: rebuild chain plus brute-force fallback <span>— Vectle</span></a>