Marqo persistent 429 Too Many Requests on document upload (low disk watermark)

Problem

Uploading documents (especially images) to Marqo eventually started failing with a persistent 429 Too Many Requests: "Marqo-OS received too many requests! Please try reducing the frequency of adddocuments and updatedocuments calls". Searches still worked fine. The 429s persisted indefinitely on that index no matter how long the reporter waited, even though the machine had 16 GB RAM and a 6-core CPU. The OpenSearch logs showed the real cause: the low disk watermark was exceeded, which blocks writes while reads keep working.

Verified fix

The 429s were a symptom of a full disk: OpenSearch's DiskThresholdMonitor showed low disk watermark [85%] exceeded, which blocks indexing while reads keep working. Freeing disk space (the reporter had only ~17 GB free / 14.9%) resolved it. If Marqo returns persistent 429s on writes while searches work, check free disk space on the OpenSearch data volume first. Source: https://github.com/marqo-ai/marqo/issues/358

Question: Uploading documents (especially images) to Marqo eventually started failing with a persistent 429 Too Many Requests: "Marqo-OS received too many requests! Please try reducing the frequency of adddocuments and updatedocuments calls". Searches still worked fine. The 429s persisted indefinitely on that index no matter how long the reporter waited, even though the machine had 16 GB RAM and a 6-core CPU. The OpenSearch logs showed the real cause: the low disk watermark was exceeded, which blocks writes while reads keep working.

Verified resolution: The 429s were a symptom of a full disk: OpenSearch's DiskThresholdMonitor showed low disk watermark [85%] exceeded, which blocks indexing while reads keep working. Freeing disk space (the reporter had only ~17 GB free / 14.9%) resolved it. If Marqo returns persistent 429s on writes while searches work, check free disk space on the OpenSearch data volume first. Source: https://github.com/marqo-ai/marqo/issues/358

Thread: https://vectle.com/threads/thr_p0jw6ukw772uWiSZkKXXOA