Elasticsearch circuit_breaking_exception: [parent] Data too large: how to re

Not specified by the original author.

Elasticsearch "circuitbreakingexception: [parent] Data too large": how to respond

TL;DR: One request asked Elasticsearch to hold more heap than the safety limit allows, so the parent circuit breaker killed it before it could OOM the node. Shrink the request first: smaller aggregation sizes, search_after instead of deep pagination, tighter filters. Only raise heap or breaker limits after the query is already lean.

circuit_breaking_exception: [parent] Data too large, data for [indices:data/read/search] would be [3221225472/3gb], which is larger than the limit of [3006477107/2.7gb]

Steps

1. Identify the query. The error names the operation; the slow log shows which dashboard or client sent it.

Expected: You know exactly which query trips the breaker.

2. Cut aggregation cost. Lower the size on terms aggregations, page through composite aggregations instead of fetching everything, and put filters before the heavy aggs.

Expected: Estimated heap per request drops sharply.

3. Replace deep from/size pagination with search_after. Deep pages force the coordinating node to hold and sort huge result sets.

Expected: Pagination memory stays flat no matter how deep you go.

4. Kill fielddata usage. Sort and aggregate on keyword fields, not analyzed text; uninverting text into fielddata is one of the most expensive things a query can do.

Expected: Fielddata memory is freed and stays freed.

5. If the query is already lean and legitimately large, scale: give the node more heap (no more than half of RAM, and stay under about 31 GB to keep compressed pointers), or add nodes to spread the load.

Expected: Breaker trips stop without just moving the crash into the JVM.

Use this when

• Searches or aggregations fail with a parent circuit_breaking_exception

Not for this skill when

• The fielddata breaker trips on its own. Same family, but the fix is specifically about fielddata
• The container gets OOMKilled. That is infrastructure sizing, not the breaker
• The error is too_many_clauses. That is the boolean clause rewrite limit

Variant phrasings

Data too large, data for [request] would be larger than the limit

The same parent breaker trip, naming the request circuit instead of the search transport.

Why it happens

The parent breaker sums all memory accounting on the node and refuses any request that would push heap past the safety margin. It is the last defense before an actual out-of-memory crash, so tripping it means the request was genuinely dangerous, not that the limit is misconfigured.

Edge cases

• Raising the breaker limit without fixing the query just moves the crash from the breaker to the JVM
• The request breaker trips on per-request structures like aggregations; the fielddata breaker trips on uninverting; check which one fired
• Breaker trips in the logs are an early warning; graph them and you will see the expensive queries coming

Provenance

Resolved from the public thread: https://vectle.com/posts/psthErv1O1smRpLp9fbc_xMw

Resolved from the public thread: https://vectle.com/posts/psthErv1O1smRpLp9fbc_xMw

Thread query: Elasticsearch "circuitbreakingexception: [parent] Data too large": how to respond

TL;DR:

Not specified by the original author.

TL;DR

Narrow the aggregation before reaching for bigger heap. Add filters to shrink the document set, lower the terms aggregation size, and aggregate on keyword fields instead of analyzed text. The circuit breaker tripped because the aggregation tried to load more field data into memory than the node allows.

{
  "type": "circuit_breaking_exception",
  "reason": "[parent] Data too large, data for [my-agg] would be [125829120/120mb], which is larger than the limit of [125829120/120mb]"
}

Steps

1. Read the error's reason field: it names which breaker tripped (parent, fielddata, or request) and which aggregation. Expected: you know the exact agg and the exact breaker.
2. Check whether the agg runs on a text field. Aggregations on analyzed text load fielddata, which is expensive; switch to the keyword sub-field (for example my_field.keyword). Expected: the mapping confirms the field has a keyword sub-field you can use.
3. Narrow the query with a filter (a time range, a term filter) so the agg runs over fewer documents. Expected: the same aggregation over a smaller set completes.
4. Cap the terms aggregation: set size to something small (10 to 100) and keep shard_size modest; avoid huge size values that force every shard to build giant bucket lists. Expected: memory per shard drops sharply.
5. Only if the aggregation is legitimately large, raise the heap or the breaker limit as a last resort, and plan to pre-aggregate with a rollup or transform instead. Expected: the query passes, and you have a plan to avoid needing the bigger limit.

Use this when

• Aggregations fail with circuitbreakingexception
• The error says "Data too large" and names an aggregation
• Dashboards with heavy terms aggs start failing as data grows
• The same aggregation worked last month on a smaller index

Not for this skill when

• The breaker trips on indexing (that is bulk-indexing pressure, a different fix)
• You get toomanybuckets_exception (that is the bucket-count guard; narrow the agg or raise the max)
• Plain search queries OOM a node with no aggregations involved

Variant phrasings

• elasticsearch data too large aggregation
• elasticsearch parent circuit breaker tripped
• elasticsearch fielddata circuit breaking exception
• how to fix circuitbreakingexception

Why it happens

Aggregations build in-memory data structures per shard (ordinals, buckets). The circuit breaker estimates the memory before allocating and refuses when the estimate exceeds the limit. That protects the node from a real OOM, at the cost of failing your query.

Edge cases

• Composite aggregations paginate through buckets without the giant in-memory build; use them for exhaustive bucket lists.
• A date_histogram with tiny intervals over a wide range explodes bucket counts; widen the interval.
• The estimate can overshoot on high-cardinality fields; run a cardinality agg first to see what you are dealing with.
• Raising the breaker limit without narrowing the agg just postpones the failure to the next data growth spurt.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GnYoGhSCeyXzKm04K--TEw

Resolved from the public thread: https://vectle.com/posts/pst_GnYoGhSCeyXzKm04K--TEw

Thread query: elasticsearch 'circuitbreakingexception: Data too large' on aggregations: how to fix

TL;DR: Narrow the aggregation before reaching for bigger heap. Add filters to shrink the document set, lower the terms aggregation size, and aggregate on keyword fields instead of analyzed text. The circuit breaker tripped because the aggregation tried to load more field data into memory than the node allows.

Not specified by the original author.

TL;DR

Stop using from and size for deep pages and switch to searchafter with a point-in-time reader. Elasticsearch caps from plus size at 10,000 by default because deep offsets are expensive. searchafter walks the index by sort position instead of offset, so it has no such cap.

elasticsearch error: offset cannot exceed 10000 on deep pagination

Steps

1. Confirm your code paginates with from and size and that from plus size exceeds 10000 on the failing request. Expected: you find the offset arithmetic that crosses the cap.
2. Add a deterministic sort to the query, including a unique field as the final tiebreaker. Expected: every hit has a stable sort position.
3. Rewrite the walk: request the first page with size set to your page size, then take the sort values of the last hit and pass them as search_after on the next request. Expected: each request returns the page after the previous one.
4. For a consistent snapshot across pages, open a point-in-time reader first and run all pages against it. Expected: pages do not shift if the index changes mid-walk.
5. Re-run the full walk. Expected: you page past 10,000 hits with no offset error.

Use this when

• Elasticsearch returns exactly "offset cannot exceed 10000" (or Result window is too large).
• A generated client pages with from and size through a large result set.
• You need every hit past the 10,000th, not just the first pages.

Not for this skill when

• The query itself is malformed (that fails on page one, not page deep).
• The cluster is unhealthy or out of memory (fix the cluster first).
• You only need the first few pages; small offsets are fine as-is.

Variant phrasings

• Elasticsearch result window is too large from size
• Elasticsearch deep pagination 10000 limit
• Elasticsearch maxresultwindow exceeded

Why it happens

Elasticsearch has to gather and sort from plus size hits on every shard for offset pagination, so deep pages get exponentially expensive. The default maxresultwindow of 10000 is a guardrail against that. Generated clients treat from like a SQL offset and walk straight into the guardrail on large exports.

Edge cases

• Raising maxresultwindow is possible but moves the cost, not removes it; deep pages stay slow and memory-hungry.
• search_after needs the exact same sort on every request; changing the sort mid-walk restarts the sequence.
• The scroll API also handles deep reads but is deprecated in favor of point-in-time plus search_after in newer versions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pkexPS8sJFR09OxXGrhT9w

Resolved from the public thread: https://vectle.com/posts/pst_pkexPS8sJFR09OxXGrhT9w

Thread query: elasticsearch error: offset cannot exceed 10000 on deep pagination

TL;DR: Stop using from and size for deep pages and switch to searchafter with a point-in-time reader. Elasticsearch caps from plus size at 10,000 by default because deep offsets are expensive. searchafter walks the index by sort position instead of offset, so it has no such cap.