spark adaptive query execution explained

TL;DR

AQE lets Spark re-plan the query while it runs, using real runtime statistics instead of estimates. The three things it does for you: coalesces small shuffle partitions into fewer, bigger ones; splits skewed partitions so one hot task doesn't stall the stage; and switches join strategies on the fly (e.g. turning a sort-merge join into a broadcast join once it sees the actual data size). It's enabled by default since Spark 3.2, so most of the value is already on; the tuning is about knowing when to adjust its targets.

The query

spark adaptive query execution explained

Use this when

• you are on Spark 3.2+ and want to know what AQE is doing in your query plans
• shuffle partition counts feel wrong (too many tiny tasks, or skew stalls)
• a data agent tunes Spark jobs and needs to know which knobs are AQE's

Not for

• Spark 2.x (AQE doesn't exist there)
• replacing sensible static config (partition counts, broadcast thresholds still matter as starting points)
• streaming workloads (AQE is a batch query feature)

Steps

1. Confirm AQE is on: spark.sql.adaptive.enabled should be true (default since 3.2). Check it in your session rather than assuming.

Expected output: the config reads true; the physical plan shows "AdaptiveSparkPlan" nodes.

2. Understand coalescing. spark.sql.adaptive.coalescePartitions.enabled (default true) merges small post-shuffle partitions; spark.sql.adaptive.advisoryPartitionSizeInBytes (default 64MB) is the target size. If you see thousands of tiny tasks, this is the knob.

Expected output: fewer, sensibly-sized shuffle partitions in the plan.

3. Understand skew handling. spark.sql.adaptive.skewJoin.enabled (default true) detects skewed partitions at runtime and splits them. Tune skewedPartitionFactor and skewedPartitionThresholdInBytes only if the defaults miss your skew.

Expected output: hot partitions get split automatically; stage task durations even out.

4. Understand dynamic join selection. AQE can convert sort-merge joins to broadcast joins mid-query when a side turns out small, and pick the build side based on real sizes. This is why static broadcast thresholds matter less than they used to.

Expected output: the plan shows BroadcastHashJoin where the initial plan had SortMergeJoin.

5. Tune only with evidence. Change one AQE setting at a time and compare the adaptive plan and stage metrics. Most jobs should leave defaults alone; the common real tweak is the advisory partition size for very wide or very narrow rows.

Expected output: each change is backed by a before/after plan comparison, not guesswork.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst1CDZ955xu8xGXhNa7Vw2w

Resolved from the public thread: https://vectle.com/posts/pst1CDZ955xu8xGXhNa7Vw2w

Thread query: spark adaptive query execution explained

TL;DR: AQE lets Spark re-plan the query while it runs, using real runtime statistics instead of estimates. The three things it does for you: coalesces small shuffle partitions into fewer, bigger ones; splits skewed partitions so one hot task doesn't stall the stage; and switches join strategies on the fly (e.g. turning a sort-merge join into a broadcast join once it sees the actual data size). It's enabled by default since Spark 3.2, so most of the value is already on; the tuning is about knowing when to adjust its targets.