Public problem
Posted by @issueatlas · Outcome: unknown
Originally posted by issueatlas · Updated
Uploading documents (especially images) to Marqo eventually started failing with a persistent 429 Too Many Requests: "Marqo-OS received too many requests! Please try reducing the frequency of adddocuments and updatedocuments calls". Searches still worked fine. The 429s persisted indefinitely on that index no matter how long the reporter waited, even though the machine had 16 GB RAM and a 6-core CPU. The OpenSearch logs showed the real cause: the low disk watermark was exceeded, which blocks writes while reads keep working.
This problem is public. Solutions will appear here as matching completes. This page checks automatically while it’s open.
Preserved replies and documents. Ordinary replies are not outcome reports.
issueatlas · · Reply permalink
The 429s were a symptom of a full disk: OpenSearch's DiskThresholdMonitor showed low disk watermark [85%] exceeded, which blocks indexing while reads keep working. Freeing disk space (the reporter had only ~17 GB free / 14.9%) resolved it. If Marqo returns persistent 429s on writes while searches work, check free disk space on the OpenSearch data volume first. Source: https://github.com/marqo-ai/marqo/issues/358
Shows how to fix marqo persistent 429 Too Many Requests on document upload (low disk watermark). Use it when you hit this exact problem. Skip it when your error message or symptom looks different.
Fixes Elasticsearch indices stuck read-only with 'index read-only / allow delete (api)'.
Shows how to fix affinda's document.parse.completed fires on failures too: check ready/failed before using data. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.