bigquery 'Resources exceeded during query execution'

TL;DR

BigQuery ran out of resources mid-query, usually because of a join explosion, a huge ORDER BY without limits, or window functions over unbounded partitions. Find the exploding stage in the query plan, then reduce the fanout: filter earlier, aggregate before joining, or break the query into staged temp tables.

bigquery 'Resources exceeded during query execution'

Use this when

• A BigQuery query fails with resources exceeded
• An agent's query works on a sample but dies on the full table
• The error mentions shuffle or memory in the details

Not for this skill when

• The error is about quota or slots (thats capacity, not the query)
• Access is denied (thats IAM)
• The SQL doesnt parse (thats syntax)

Steps

1. Open the query plan in the console and find the stage with the biggest output rows:

Console: query history \u2192 the failed job \u2192 Execution details

Expected output: one stage shows output rows far larger than its input. That stage is the explosion; everything downstream of it is a victim.

1. Check for accidental cross joins first:

-- broken: missing join condition fans out
FROM orders, customers  -- did you mean to join on customer_id?

Expected output: you find the missing ON clause. A comma join or a join on a non-unique key multiplies rows and is the top cause.

1. Push filters and aggregations before the big join:

WITH filtered AS (
  SELECT customer_id, SUM(amount) AS total
  FROM orders
  WHERE order_date BETWEEN DATE '2026-01-01' AND CURRENT_DATE
  GROUP BY customer_id
)
SELECT * FROM filtered JOIN customers USING (customer_id);

Expected output: the join inputs shrink by orders of magnitude and the query fits in resources.

1. For window functions, bound the partitions:

-- add PARTITION BY so no single partition holds the whole table
ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_date)

Expected output: memory per partition stays bounded. An unpartitioned window over a huge table is the second most common cause.

Variant phrasings

resources exceeded during shuffle

The join fanout is the problem (step 2-3). Shuffle is where BigQuery moves join data; explosions show up there first.

query worked last month

Data grew past the query's resource budget. The query was always borderline; volume pushed it over.

Why it happens

BigQuery executes in stages with per-stage memory and shuffle limits. A join that multiplies rows (bad key, missing condition) or a sort/window over an unbounded set exceeds those limits no matter how many slots you have. The error is about the query's shape, not your quota, which is why buying more slots doesnt fix it.

Edge cases

• JOIN on a key with heavy skew (one key holding most rows) exceeds limits even with correct join conditions; salt the key or use a broadcast join hint.
• SELECT * through the explosion carries every column through the shuffle; project only needed columns.
• If the query is legitimately huge, materialize intermediate results to temp tables and run it in stages.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst9vZmezVhLhPEA2X1Gl9Cg