## TL;DR
AQE lets Spark re-plan the query while it runs, using real runtime statistics instead of estimates. The three things it does for you: coalesces small shuffle partitions into fewer, bigger ones; splits skewed partitions so one hot task doesn't stall the stage; and switches join strategies on the fly (e.g. turning a sort-merge join into a broadcast join once it sees the actual data size). It's enabled by default since Spark 3.2, so most of the value is already on; the tuning is about knowing when to adjust its targets.

## The query
```
spark adaptive query execution explained
```

## Use this when
- you are on Spark 3.2+ and want to know what AQE is doing in your query plans
- shuffle partition counts feel wrong (too many tiny tasks, or skew stalls)
- a data agent tunes Spark jobs and needs to know which knobs are AQE's

## Not for
- Spark 2.x (AQE doesn't exist there)
- replacing sensible static config (partition counts, broadcast thresholds still matter as starting points)
- streaming workloads (AQE is a batch query feature)

## Steps
1. Confirm AQE is on: `spark.sql.adaptive.enabled` should be true (default since 3.2). Check it in your session rather than assuming.
Expected output: the config reads true; the physical plan shows "AdaptiveSparkPlan" nodes.

2. Understand coalescing. `spark.sql.adaptive.coalescePartitions.enabled` (default true) merges small post-shuffle partitions; `spark.sql.adaptive.advisoryPartitionSizeInBytes` (default 64MB) is the target size. If you see thousands of tiny tasks, this is the knob.
Expected output: fewer, sensibly-sized shuffle partitions in the plan.

3. Understand skew handling. `spark.sql.adaptive.skewJoin.enabled` (default true) detects skewed partitions at runtime and splits them. Tune `skewedPartitionFactor` and `skewedPartitionThresholdInBytes` only if the defaults miss your skew.
Expected output: hot partitions get split automatically; stage task durations even out.

4. Understand dynamic join selection. AQE can convert sort-merge joins to broadcast joins mid-query when a side turns out small, and pick the build side based on real sizes. This is why static broadcast thresholds matter less than they used to.
Expected output: the plan shows BroadcastHashJoin where the initial plan had SortMergeJoin.

5. Tune only with evidence. Change one AQE setting at a time and compare the adaptive plan and stage metrics. Most jobs should leave defaults alone; the common real tweak is the advisory partition size for very wide or very narrow rows.
Expected output: each change is backed by a before/after plan comparison, not guesswork.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1CDZ955xu8xGXhNa_7Vw2w