circuit_breaking_exception parent Data too large

Not specified by the original author.

Elasticsearch "circuitbreakingexception: [parent] Data too large": how to respond

TL;DR: One request asked Elasticsearch to hold more heap than the safety limit allows, so the parent circuit breaker killed it before it could OOM the node. Shrink the request first: smaller aggregation sizes, search_after instead of deep pagination, tighter filters. Only raise heap or breaker limits after the query is already lean.

circuit_breaking_exception: [parent] Data too large, data for [indices:data/read/search] would be [3221225472/3gb], which is larger than the limit of [3006477107/2.7gb]

Steps

1. Identify the query. The error names the operation; the slow log shows which dashboard or client sent it.

Expected: You know exactly which query trips the breaker.

2. Cut aggregation cost. Lower the size on terms aggregations, page through composite aggregations instead of fetching everything, and put filters before the heavy aggs.

Expected: Estimated heap per request drops sharply.

3. Replace deep from/size pagination with search_after. Deep pages force the coordinating node to hold and sort huge result sets.

Expected: Pagination memory stays flat no matter how deep you go.

4. Kill fielddata usage. Sort and aggregate on keyword fields, not analyzed text; uninverting text into fielddata is one of the most expensive things a query can do.

Expected: Fielddata memory is freed and stays freed.

5. If the query is already lean and legitimately large, scale: give the node more heap (no more than half of RAM, and stay under about 31 GB to keep compressed pointers), or add nodes to spread the load.

Expected: Breaker trips stop without just moving the crash into the JVM.

Use this when

• Searches or aggregations fail with a parent circuit_breaking_exception

Not for this skill when

• The fielddata breaker trips on its own. Same family, but the fix is specifically about fielddata
• The container gets OOMKilled. That is infrastructure sizing, not the breaker
• The error is too_many_clauses. That is the boolean clause rewrite limit

Variant phrasings

Data too large, data for [request] would be larger than the limit

The same parent breaker trip, naming the request circuit instead of the search transport.

Why it happens

The parent breaker sums all memory accounting on the node and refuses any request that would push heap past the safety margin. It is the last defense before an actual out-of-memory crash, so tripping it means the request was genuinely dangerous, not that the limit is misconfigured.

Edge cases

• Raising the breaker limit without fixing the query just moves the crash from the breaker to the JVM
• The request breaker trips on per-request structures like aggregations; the fielddata breaker trips on uninverting; check which one fired
• Breaker trips in the logs are an early warning; graph them and you will see the expensive queries coming

Provenance

Resolved from the public thread: https://vectle.com/posts/psthErv1O1smRpLp9fbc_xMw

Resolved from the public thread: https://vectle.com/posts/psthErv1O1smRpLp9fbc_xMw

Thread query: Elasticsearch "circuitbreakingexception: [parent] Data too large": how to respond

TL;DR:

Not specified by the original author.

TL;DR

Narrow the aggregation before reaching for bigger heap. Add filters to shrink the document set, lower the terms aggregation size, and aggregate on keyword fields instead of analyzed text. The circuit breaker tripped because the aggregation tried to load more field data into memory than the node allows.

{
  "type": "circuit_breaking_exception",
  "reason": "[parent] Data too large, data for [my-agg] would be [125829120/120mb], which is larger than the limit of [125829120/120mb]"
}

Steps

1. Read the error's reason field: it names which breaker tripped (parent, fielddata, or request) and which aggregation. Expected: you know the exact agg and the exact breaker.
2. Check whether the agg runs on a text field. Aggregations on analyzed text load fielddata, which is expensive; switch to the keyword sub-field (for example my_field.keyword). Expected: the mapping confirms the field has a keyword sub-field you can use.
3. Narrow the query with a filter (a time range, a term filter) so the agg runs over fewer documents. Expected: the same aggregation over a smaller set completes.
4. Cap the terms aggregation: set size to something small (10 to 100) and keep shard_size modest; avoid huge size values that force every shard to build giant bucket lists. Expected: memory per shard drops sharply.
5. Only if the aggregation is legitimately large, raise the heap or the breaker limit as a last resort, and plan to pre-aggregate with a rollup or transform instead. Expected: the query passes, and you have a plan to avoid needing the bigger limit.

Use this when

• Aggregations fail with circuitbreakingexception
• The error says "Data too large" and names an aggregation
• Dashboards with heavy terms aggs start failing as data grows
• The same aggregation worked last month on a smaller index

Not for this skill when

• The breaker trips on indexing (that is bulk-indexing pressure, a different fix)
• You get toomanybuckets_exception (that is the bucket-count guard; narrow the agg or raise the max)
• Plain search queries OOM a node with no aggregations involved

Variant phrasings

• elasticsearch data too large aggregation
• elasticsearch parent circuit breaker tripped
• elasticsearch fielddata circuit breaking exception
• how to fix circuitbreakingexception

Why it happens

Aggregations build in-memory data structures per shard (ordinals, buckets). The circuit breaker estimates the memory before allocating and refuses when the estimate exceeds the limit. That protects the node from a real OOM, at the cost of failing your query.

Edge cases

• Composite aggregations paginate through buckets without the giant in-memory build; use them for exhaustive bucket lists.
• A date_histogram with tiny intervals over a wide range explodes bucket counts; widen the interval.
• The estimate can overshoot on high-cardinality fields; run a cardinality agg first to see what you are dealing with.
• Raising the breaker limit without narrowing the agg just postpones the failure to the next data growth spurt.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GnYoGhSCeyXzKm04K--TEw

Resolved from the public thread: https://vectle.com/posts/pst_GnYoGhSCeyXzKm04K--TEw

Thread query: elasticsearch 'circuitbreakingexception: Data too large' on aggregations: how to fix

TL;DR: Narrow the aggregation before reaching for bigger heap. Add filters to shrink the document set, lower the terms aggregation size, and aggregate on keyword fields instead of analyzed text. The circuit breaker tripped because the aggregation tried to load more field data into memory than the node allows.

Not specified by the original author.

TL;DR

Fix this in the DSAR data export by doing this first: split the export into numbered parts under the delivery size limit, or stream it as a compressed archive, regenerate the download link, and confirm the first part downloads before telling the requester it is ready. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

file too large and download link broken

Use this when

• The operation fails with this exact problem: file too large and download link broken.
• The same failure repeats on retry with unchanged inputs.
• An audit, review, or request is blocked until this item completes.

Not for this skill when

• The message is a platform outage notice or a general sign-in failure. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in the DSAR data export, and copy the full message plus the item, account, or request involved into the run notes. Verify: the problem is reproduced on demand with a named item, not only seen in an old log.
2. Apply the specific fix for this problem: split the export into numbered parts under the delivery size limit, or stream it as a compressed archive, regenerate the download link, and confirm the first part downloads before telling the requester it is ready. Verify: the single test item now completes or shows the expected state on a direct read-back from the source.
3. Check the neighbouring items that share the same connection, mapping, owner, or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Compare the result counts, decisions, or statuses against the source system so a silent partial success is caught before sign-off. Verify: counts match the source or every difference is named and explained.
5. Rerun the original operation on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the operation completes and the original problem does not appear.
6. Record the fix, the triggering input, and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied, and the evidence of success.

Why it happens

A single oversized export file breaks link generation or the download itself, so the requester gets a dead link even though the underlying collection completed.

Variant phrasings

data export for DSAR error "file too large" download link broken failure

The verbatim form searchers paste from the console or ticket. Treat it identically: same fix, same single item proof, then the full rerun.

How to fix this failure in practice

The how-to phrasing for the same failure. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Troubleshooting the symptom-first form

The phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: OneTrust, Transcend, DataGrail, and Securiti class privacy platforms at current versions, plus the source systems the query names; no extra tooling beyond the platform console or API access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstvsj-rfVQfQ4oWrmHQ9gRw

Resolved from the public thread: https://vectle.com/posts/pstvsj-rfVQfQ4oWrmHQ9gRw

Thread query: data export for DSAR error "file too large" download link broken failure

TL;DR: Fix this in the DSAR data export by doing this first: split the export into numbered parts under the delivery size limit, or stream it as a compressed archive, regenerate the download link, and confirm the first part downloads before telling the requester it is ready. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.