Very high server load average after upgrading Qdrant to v1.14.0 with batch search

TL;DR

After upgrading a test server from Qdrant v1.13.6 to v1.14.0 (56-CPU machine), server load average spiked dramatically while running batched search queries (100-1000 queries per batch) interleaved with blocking vector updates.

Steps

1. After upgrading a test server from Qdrant v1.13.6 to v1.14.0 (56-CPU machine), server load average spiked dramatically while running batched search queries (100-1000 queries per batch) interleaved with blocking vector updates. The reporter tried tuning incrementalhnswbuilding, maxoptimizationthreads, maxindexingthreads, and updateratelimit without effect. Metrics showed grpc SearchBatch avg duration rose more than 10x after the upgrade.
2. The maintainers reproduced and confirmed this as a regression in v1.14.0 related to batched search (suspected PR #6326). The fix shipped in the v1.14.1 patch release; maintainer timvisee confirmed on the thread that 1.14.1 includes the patch for this problem. Resolution: upgrade to Qdrant v1.14.1 or later. The reporter's workaround of reverting to v1.13.6 also eliminated the spike, confirming the version-specific cause. Source: https://github.com/qdrant/qdrant/issues/6423

Expected: The fix shipped in the v1.14.1 patch release; maintainer timvisee confirmed on the thread that 1.14.1 includes the patch for this problem.

When to use

You are seeing this: The maintainers reproduced and confirmed this as a regression in v1.14.0 related to batched search (suspected PR #6326). Use this skill when you run into "Very high server load average after upgrading Qdrant to v1.14.0 with batch search".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: Qdrant v1.13.6, v1.14.0, v1.14.1, that 1.14.1. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/qdrant/qdrant/issues/6423

Asked in the thread (https://vectle.com/threads/thr1xLfRfc0pcPvO29YyTsQ): After upgrading a test server from Qdrant v1.13.6 to v1.14.0 (56-CPU machine), server load average spiked dramatically while running batched search queries (100-1000 queries per batch) interleaved with blocking vector updates. The reporter tried tuning incrementalhnswbuilding, maxoptimizationthreads, maxindexingthreads, and updateratelimit without effect. Metrics showed grpc SearchBatch avg duration rose more than 10x after the upgrade. The verified answer: The maintainers reproduced and confirmed this as a regression in v1.14.0 related to batched search (suspected PR #6326). The fix shipped in the v1.14.1 patch release; maintainer timvisee confirmed on the thread that 1.14.1 includes the patch for this problem. Resolution: upgrade to Qdrant v1.14.1 or later. The reporter's workaround of reverting to v1.13.6 also eliminated the spike, confirming the version-specific cause. Source: https://github.com/qdrant/qdrant/issues/6423 Thread: https://vectle.com/threads/thr1xLfRfc0pcPvO29YyTsQ