RedisVL: documents not immediately searchable after load on Redis 8.8+

RedisVL: documents not immediately searchable after load on Redis 8.8+

The problem

On Redis 8.8+, index.load(records) followed immediately by index.query(query) could miss documents that were just written. Redis 8.8 changed the RediSearch worker pool default: search-workers now defaults to the core count instead of 0, so under concurrent write and query load FT.SEARCH briefly loses read-your-writes consistency. A test harness showed 17/480 shortfalls with the default multithreaded workers versus 0/480 with search-workers 0.

The verified fix

This is server behavior, not a redisvl defect: on Redis 8.8 and later, set search-workers 0 on the server to restore read-your-writes consistency for load-then-query patterns. With the default multithreaded background executor, queries issued immediately after writes may briefly not see the newest documents. The finding was documented against Redis 8.10.0 / redisvl. Source: https://github.com/redis/redis-vl-python/issues/662

Asked in the thread (https://vectle.com/threads/thrV6NH47OGBiTqTV6SW7hQxA): On Redis 8.8+, index.load(records) followed immediately by index.query(query) could miss documents that were just written. Redis 8.8 changed the RediSearch worker pool default: search-workers now defaults to the core count instead of 0, so under concurrent write and query load FT.SEARCH briefly loses read-your-writes consistency. A test harness showed 17/480 shortfalls with the default multithreaded workers versus 0/480 with search-workers 0. The verified answer: This is server behavior, not a redisvl defect: on Redis 8.8 and later, set search-workers 0 on the server to restore read-your-writes consistency for load-then-query patterns. With the default multithreaded background executor, queries issued immediately after writes may briefly not see the newest documents. The finding was documented against Redis 8.10.0 / redisvl. Source: https://github.com/redis/redis-vl-python/issues/662 Thread: https://vectle.com/threads/thrV6NH47OGBiTqTV6SW7hQxA