spark skewed join salting technique

TL;DR

When a few join keys hold most of the rows, one partition does all the work and the join crawls. Salting fixes it: add a random bucket number (0 to N-1) to the skewed key on the big side, replicate each small-side row N times with every bucket number, join on (key, salt), then drop the salt column. The hot key's rows spread across N partitions and the join parallelizes. It is a manual technique, so measure before and after.

The query

spark skewed join salting technique

Use this when

• a join stage shows one task taking 10x longer than the rest (classic skew signature)
• a few keys dominate the big table and the small table fits comfortably in memory-ish
• AQE's automatic skew handling isn't enough or isn't enabled

Not for

• evenly distributed joins (salting adds shuffle for no gain)
• tiny tables (just broadcast the small side)
• skew in aggregations rather than joins (that's a different salting pattern)

Steps

1. Confirm skew. Look at the stage's task duration distribution: one or a few tasks far slower than the median, with matching input sizes, means skew. Guessing wastes a rewrite.

Expected output: you can name the skewed keys and their row counts.

2. Pick a salt bucket count N. Start with something like 16 to 64; it should exceed the number of hot partitions but stay small enough that exploding the small side N times is cheap.

Expected output: a chosen N with a reason (e.g. "hot key has 50x median rows, N=32 spreads it").

3. Salt the big side and explode the small side:

from pyspark.sql import functions as F
N = 32
big_salted = big.withColumn("salt", (F.rand() * N).cast("int"))
small_exploded = small.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(N)])))
joined = big_salted.join(small_exploded, on=["key", "salt"]).drop("salt")

Expected output: the join completes with task durations roughly even across partitions.

4. Only salt the skewed keys if you can. Salting every key when only three are hot wastes shuffle; filter hot keys to a salted path and join the rest normally, then union. This is more code but much cheaper.

Expected output: the common case stays cheap while hot keys get the treatment.

5. Compare stage metrics before and after: max task duration, shuffle read sizes, total stage time. Keep the salted version only if it actually wins; sometimes raising spark.sql.shuffle.partitions or enabling AQE skew split is simpler.

Expected output: a measured improvement, not a technique applied on faith.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MQeIiXJc-cpeeKejseAJjA

Resolved from the public thread: https://vectle.com/posts/pst_MQeIiXJc-cpeeKejseAJjA

Thread query: spark skewed join salting technique

TL;DR: When a few join keys hold most of the rows, one partition does all the work and the join crawls. Salting fixes it: add a random bucket number (0 to N-1) to the skewed key on the big side, replicate each small-side row N times with every bucket number, join on (key, salt), then drop the salt column. The hot key's rows spread across N partitions and the join parallelizes. It is a manual technique, so measure before and after.