elasticsearch error: offset cannot exceed 10000 on deep pagination

TL;DR

Stop using from and size for deep pages and switch to searchafter with a point-in-time reader. Elasticsearch caps from plus size at 10,000 by default because deep offsets are expensive. searchafter walks the index by sort position instead of offset, so it has no such cap.

elasticsearch error: offset cannot exceed 10000 on deep pagination

Steps

1. Confirm your code paginates with from and size and that from plus size exceeds 10000 on the failing request. Expected: you find the offset arithmetic that crosses the cap.
2. Add a deterministic sort to the query, including a unique field as the final tiebreaker. Expected: every hit has a stable sort position.
3. Rewrite the walk: request the first page with size set to your page size, then take the sort values of the last hit and pass them as search_after on the next request. Expected: each request returns the page after the previous one.
4. For a consistent snapshot across pages, open a point-in-time reader first and run all pages against it. Expected: pages do not shift if the index changes mid-walk.
5. Re-run the full walk. Expected: you page past 10,000 hits with no offset error.

Use this when

• Elasticsearch returns exactly "offset cannot exceed 10000" (or Result window is too large).
• A generated client pages with from and size through a large result set.
• You need every hit past the 10,000th, not just the first pages.

Not for this skill when

• The query itself is malformed (that fails on page one, not page deep).
• The cluster is unhealthy or out of memory (fix the cluster first).
• You only need the first few pages; small offsets are fine as-is.

Variant phrasings

• Elasticsearch result window is too large from size
• Elasticsearch deep pagination 10000 limit
• Elasticsearch maxresultwindow exceeded

Why it happens

Elasticsearch has to gather and sort from plus size hits on every shard for offset pagination, so deep pages get exponentially expensive. The default maxresultwindow of 10000 is a guardrail against that. Generated clients treat from like a SQL offset and walk straight into the guardrail on large exports.

Edge cases

• Raising maxresultwindow is possible but moves the cost, not removes it; deep pages stay slow and memory-hungry.
• search_after needs the exact same sort on every request; changing the sort mid-walk restarts the sequence.
• The scroll API also handles deep reads but is deprecated in favor of point-in-time plus search_after in newer versions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pkexPS8sJFR09OxXGrhT9w