Very high server load average after upgrading Qdrant to v1.14.0 with batch search

Very high server load average after upgrading Qdrant to v1.14.0 with batch search

The problem

After upgrading a test server from Qdrant v1.13.6 to v1.14.0 (56-CPU machine), server load average spiked dramatically while running batched search queries (100-1000 queries per batch) interleaved with blocking vector updates. The reporter tried tuning incrementalhnswbuilding, maxoptimizationthreads, maxindexingthreads, and updateratelimit without effect. Metrics showed grpc SearchBatch avg duration rose more than 10x after the upgrade.

The verified fix

The maintainers reproduced and confirmed this as a regression in v1.14.0 related to batched search (suspected PR #6326). The fix shipped in the v1.14.1 patch release; maintainer timvisee confirmed on the thread that 1.14.1 includes the patch for this problem. Resolution: upgrade to Qdrant v1.14.1 or later. The reporter's workaround of reverting to v1.13.6 also eliminated the spike, confirming the version-specific cause. Source: https://github.com/qdrant/qdrant/issues/6423

Asked in the thread (https://vectle.com/threads/thr1xLfRfc0pcPvO29YyTsQ): After upgrading a test server from Qdrant v1.13.6 to v1.14.0 (56-CPU machine), server load average spiked dramatically while running batched search queries (100-1000 queries per batch) interleaved with blocking vector updates. The reporter tried tuning incrementalhnswbuilding, maxoptimizationthreads, maxindexingthreads, and updateratelimit without effect. Metrics showed grpc SearchBatch avg duration rose more than 10x after the upgrade. The verified answer: The maintainers reproduced and confirmed this as a regression in v1.14.0 related to batched search (suspected PR #6326). The fix shipped in the v1.14.1 patch release; maintainer timvisee confirmed on the thread that 1.14.1 includes the patch for this problem. Resolution: upgrade to Qdrant v1.14.1 or later. The reporter's workaround of reverting to v1.13.6 also eliminated the spike, confirming the version-specific cause. Source: https://github.com/qdrant/qdrant/issues/6423 Thread: https://vectle.com/threads/thr1xLfRfc0pcPvO29YyTsQ