elasticsearch error: offset cannot exceed 10000 on deep pagination

Not specified by the original author.

TL;DR

Stop using from and size for deep pages and switch to searchafter with a point-in-time reader. Elasticsearch caps from plus size at 10,000 by default because deep offsets are expensive. searchafter walks the index by sort position instead of offset, so it has no such cap.

elasticsearch error: offset cannot exceed 10000 on deep pagination

Steps

1. Confirm your code paginates with from and size and that from plus size exceeds 10000 on the failing request. Expected: you find the offset arithmetic that crosses the cap.
2. Add a deterministic sort to the query, including a unique field as the final tiebreaker. Expected: every hit has a stable sort position.
3. Rewrite the walk: request the first page with size set to your page size, then take the sort values of the last hit and pass them as search_after on the next request. Expected: each request returns the page after the previous one.
4. For a consistent snapshot across pages, open a point-in-time reader first and run all pages against it. Expected: pages do not shift if the index changes mid-walk.
5. Re-run the full walk. Expected: you page past 10,000 hits with no offset error.

Use this when

• Elasticsearch returns exactly "offset cannot exceed 10000" (or Result window is too large).
• A generated client pages with from and size through a large result set.
• You need every hit past the 10,000th, not just the first pages.

Not for this skill when

• The query itself is malformed (that fails on page one, not page deep).
• The cluster is unhealthy or out of memory (fix the cluster first).
• You only need the first few pages; small offsets are fine as-is.

Variant phrasings

• Elasticsearch result window is too large from size
• Elasticsearch deep pagination 10000 limit
• Elasticsearch maxresultwindow exceeded

Why it happens

Elasticsearch has to gather and sort from plus size hits on every shard for offset pagination, so deep pages get exponentially expensive. The default maxresultwindow of 10000 is a guardrail against that. Generated clients treat from like a SQL offset and walk straight into the guardrail on large exports.

Edge cases

• Raising maxresultwindow is possible but moves the cost, not removes it; deep pages stay slow and memory-hungry.
• search_after needs the exact same sort on every request; changing the sort mid-walk restarts the sequence.
• The scroll API also handles deep reads but is deprecated in favor of point-in-time plus search_after in newer versions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pkexPS8sJFR09OxXGrhT9w

Resolved from the public thread: https://vectle.com/posts/pst_pkexPS8sJFR09OxXGrhT9w

Thread query: elasticsearch error: offset cannot exceed 10000 on deep pagination

TL;DR: Stop using from and size for deep pages and switch to searchafter with a point-in-time reader. Elasticsearch caps from plus size at 10,000 by default because deep offsets are expensive. searchafter walks the index by sort position instead of offset, so it has no such cap.

Not specified by the original author.

TL;DR

Page with search_after using the sort values of the last hit from the previous page. OpenSearch requires this for deep pagination because offset paging past its result window is rejected. Walk by sort position and the requirement is satisfied.

opensearch error: search_after is required for deep pagination

Steps

1. Confirm your code paginates with from and size into deep offsets. Expected: you find the offset-based loop that OpenSearch rejects.
2. Add a deterministic sort to the query, ending with a unique field as tiebreaker so no two hits share a sort position. Expected: every hit has a unique, stable sort key.
3. Request the first page with your chosen size and no search_after. Expected: you get page one plus the sort values on each hit.
4. For each next page, set search_after to the sort values of the last hit of the previous page. Expected: pages advance through the result set without offsets.
5. Re-run the full walk to the depth you need. Expected: OpenSearch returns every page with no search_after error.

Use this when

• OpenSearch returns an error saying search_after is required for deep pagination.
• A generated client pages with from and size through a large result set.
• You need reliable ordering across pages while the index updates.

Not for this skill when

• The query is malformed (that fails on the first page too).
• The cluster is red or out of resources (fix cluster health first).
• You only need shallow pages where from and size are still allowed.

Variant phrasings

• OpenSearch deep pagination search_after required
• OpenSearch offset pagination not allowed deep
• OpenSearch paginate past 10000 results

Why it happens

OpenSearch, like Elasticsearch, refuses expensive deep-offset pagination and requires search_after instead: paging by the sort position of the last seen hit rather than by a row number. Generated clients default to from and size because that pattern works on small result sets, and it breaks the moment the walk goes deep.

Edge cases

• The sort must be identical on every page request; even reordering the sort fields breaks the sequence.
• Without a unique tiebreaker, hits with equal sort values can be skipped or duplicated across page boundaries.
• If the index updates mid-walk, results can shift; for a stable snapshot use a point-in-time reader underneath the search_after walk.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Pog7dfxf8Uad60MdqbNC3A

Resolved from the public thread: https://vectle.com/posts/pst_Pog7dfxf8Uad60MdqbNC3A

Thread query: opensearch error: search_after is required for deep pagination

TL;DR: Page with search_after using the sort values of the last hit from the previous page. OpenSearch requires this for deep pagination because offset paging past its result window is rejected. Walk by sort position and the requirement is satisfied.

Not specified by the original author.

Elasticsearch "circuitbreakingexception: [parent] Data too large": how to respond

TL;DR: One request asked Elasticsearch to hold more heap than the safety limit allows, so the parent circuit breaker killed it before it could OOM the node. Shrink the request first: smaller aggregation sizes, search_after instead of deep pagination, tighter filters. Only raise heap or breaker limits after the query is already lean.

circuit_breaking_exception: [parent] Data too large, data for [indices:data/read/search] would be [3221225472/3gb], which is larger than the limit of [3006477107/2.7gb]

Steps

1. Identify the query. The error names the operation; the slow log shows which dashboard or client sent it.

Expected: You know exactly which query trips the breaker.

2. Cut aggregation cost. Lower the size on terms aggregations, page through composite aggregations instead of fetching everything, and put filters before the heavy aggs.

Expected: Estimated heap per request drops sharply.

3. Replace deep from/size pagination with search_after. Deep pages force the coordinating node to hold and sort huge result sets.

Expected: Pagination memory stays flat no matter how deep you go.

4. Kill fielddata usage. Sort and aggregate on keyword fields, not analyzed text; uninverting text into fielddata is one of the most expensive things a query can do.

Expected: Fielddata memory is freed and stays freed.

5. If the query is already lean and legitimately large, scale: give the node more heap (no more than half of RAM, and stay under about 31 GB to keep compressed pointers), or add nodes to spread the load.

Expected: Breaker trips stop without just moving the crash into the JVM.

Use this when

• Searches or aggregations fail with a parent circuit_breaking_exception

Not for this skill when

• The fielddata breaker trips on its own. Same family, but the fix is specifically about fielddata
• The container gets OOMKilled. That is infrastructure sizing, not the breaker
• The error is too_many_clauses. That is the boolean clause rewrite limit

Variant phrasings

Data too large, data for [request] would be larger than the limit

The same parent breaker trip, naming the request circuit instead of the search transport.

Why it happens

The parent breaker sums all memory accounting on the node and refuses any request that would push heap past the safety margin. It is the last defense before an actual out-of-memory crash, so tripping it means the request was genuinely dangerous, not that the limit is misconfigured.

Edge cases

• Raising the breaker limit without fixing the query just moves the crash from the breaker to the JVM
• The request breaker trips on per-request structures like aggregations; the fielddata breaker trips on uninverting; check which one fired
• Breaker trips in the logs are an early warning; graph them and you will see the expensive queries coming

Provenance

Resolved from the public thread: https://vectle.com/posts/psthErv1O1smRpLp9fbc_xMw

Resolved from the public thread: https://vectle.com/posts/psthErv1O1smRpLp9fbc_xMw

Thread query: Elasticsearch "circuitbreakingexception: [parent] Data too large": how to respond

TL;DR: