## TL;DR
When a few join keys hold most of the rows, one partition does all the work and the join crawls. Salting fixes it: add a random bucket number (0 to N-1) to the skewed key on the big side, replicate each small-side row N times with every bucket number, join on `(key, salt)`, then drop the salt column. The hot key's rows spread across N partitions and the join parallelizes. It is a manual technique, so measure before and after.

## The query
```
spark skewed join salting technique
```

## Use this when
- a join stage shows one task taking 10x longer than the rest (classic skew signature)
- a few keys dominate the big table and the small table fits comfortably in memory-ish
- AQE's automatic skew handling isn't enough or isn't enabled

## Not for
- evenly distributed joins (salting adds shuffle for no gain)
- tiny tables (just broadcast the small side)
- skew in aggregations rather than joins (that's a different salting pattern)

## Steps
1. Confirm skew. Look at the stage's task duration distribution: one or a few tasks far slower than the median, with matching input sizes, means skew. Guessing wastes a rewrite.
Expected output: you can name the skewed keys and their row counts.

2. Pick a salt bucket count N. Start with something like 16 to 64; it should exceed the number of hot partitions but stay small enough that exploding the small side N times is cheap.
Expected output: a chosen N with a reason (e.g. "hot key has 50x median rows, N=32 spreads it").

3. Salt the big side and explode the small side:
```python
from pyspark.sql import functions as F
N = 32
big_salted = big.withColumn("salt", (F.rand() * N).cast("int"))
small_exploded = small.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(N)])))
joined = big_salted.join(small_exploded, on=["key", "salt"]).drop("salt")
```
Expected output: the join completes with task durations roughly even across partitions.

4. Only salt the skewed keys if you can. Salting every key when only three are hot wastes shuffle; filter hot keys to a salted path and join the rest normally, then union. This is more code but much cheaper.
Expected output: the common case stays cheap while hot keys get the treatment.

5. Compare stage metrics before and after: max task duration, shuffle read sizes, total stage time. Keep the salted version only if it actually wins; sometimes raising `spark.sql.shuffle.partitions` or enabling AQE skew split is simpler.
Expected output: a measured improvement, not a technique applied on faith.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MQeIiXJc-cpeeKejseAJjA