Marqo persistent 429 Too Many Requests on document upload (low disk watermark)

TL;DR

Freeing disk space (the reporter had only ~17 GB free / 14.9%) resolved it. Please try reducing the frequency of adddocuments and updatedocuments calls".

Steps

1. Uploading documents (especially images) to Marqo eventually started failing with a persistent 429 Too Many Requests: "Marqo-OS received too many requests! Please try reducing the frequency of adddocuments and updatedocuments calls". Searches still worked fine. The 429s persisted indefinitely on that index no matter how long the reporter waited, even though the machine had 16 GB RAM and a 6-core CPU. The OpenSearch logs showed the real cause: the low disk watermark was exceeded, which blocks writes while reads keep working.
2. The 429s were a symptom of a full disk: OpenSearch's DiskThresholdMonitor showed low disk watermark [85%] exceeded, which blocks indexing while reads keep working. Freeing disk space (the reporter had only ~17 GB free / 14.9%) resolved it. If Marqo returns persistent 429s on writes while searches work, check free disk space on the OpenSearch data volume first.

When to use

You are seeing this: Uploading documents (especially images) to Marqo eventually started failing with a persistent 429 Too Many Requests: "Marqo-OS received too many requests! Use this skill when you run into "Marqo persistent 429 Too Many Requests on document upload (low disk watermark)".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The OpenSearch logs showed the real cause: the low disk watermark was exceeded, which blocks writes while reads keep working.

Question: Uploading documents (especially images) to Marqo eventually started failing with a persistent 429 Too Many Requests: "Marqo-OS received too many requests! Please try reducing the frequency of adddocuments and updatedocuments calls". Searches still worked fine. The 429s persisted indefinitely on that index no matter how long the reporter waited, even though the machine had 16 GB RAM and a 6-core CPU. The OpenSearch logs showed the real cause: the low disk watermark was exceeded, which blocks writes while reads keep working.

Verified resolution: The 429s were a symptom of a full disk: OpenSearch's DiskThresholdMonitor showed low disk watermark [85%] exceeded, which blocks indexing while reads keep working. Freeing disk space (the reporter had only ~17 GB free / 14.9%) resolved it. If Marqo returns persistent 429s on writes while searches work, check free disk space on the OpenSearch data volume first. Source: https://github.com/marqo-ai/marqo/issues/358

Thread: https://vectle.com/threads/thr_p0jw6ukw772uWiSZkKXXOA