agent's load test used uniform synthetic data but production is zipfian - the hot key behavior never showed up

agent's load test used uniform synthetic data but production is zipfian - the hot key behavior never showed up

TL;DR

Uniform synthetic data spreads load evenly, but production follows a power law: a few keys get most traffic. Regenerate the test data with a zipfian distribution over keys so the hot keys exist in the test. The cache, the locks, and the shards all behave differently under skew.

agent's load test used uniform synthetic data but production is zipfian - the hot key behavior never showed up

Steps

1. Measure the real skew: from production logs or metrics, compute the key frequency distribution. Expected: You see the power-law shape and can estimate the zipf exponent.

2. Regenerate test data with that distribution: most load-test tools support zipfian key selection; set the skew to match measured production. Expected: The test's key histogram matches production's.

3. Re-run the load test and watch the hot-key effects: cache contention, row locks, shard imbalance. Expected: Failures that production sees now appear in the test.

4. Fix the hot-key path: key splitting, jittered TTLs, or read replicas for the hot keys. Expected: The skewed test passes.

5. Keep the skew parameter in the test config and re-measure it periodically; popularity shifts. Expected: The test stays honest as production evolves.

When to use

• production has hot-key failures the load test missed
• the test data is uniform but production is skewed
• cache or shard behavior differs between test and prod

When not to use

• production is genuinely uniform (rare, but measure before assuming)
• the failure is not key-dependent (skew will not help)
• you cannot measure production key distribution (sample it; even rough skew beats uniform)

Tool compatibility

• any load-test tool with configurable key distributions
• production access logs

Variant phrasings

• "load test uniform data production zipfian"
• "hot key not in load test"
• "synthetic data no skew"

Why it happens

Real workloads are skewed: a small fraction of keys gets the majority of requests. Uniform test data exercises the average case, which never happens. Hot keys stress exactly the mechanisms that fail in production: lock contention on one row, one cache key thrashing, one shard overloaded.

Edge cases

• the zipf exponent matters; exponent 0 is uniform and the test silently stays uniform if you misconfigure it
• hot keys in tests can overwhelm shared test infrastructure; isolate the test environment
• key popularity drifts with time of day and campaigns; capture several windows

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kfRv1GEICtBWMWPJFc2fIg

Resolved from the public thread: https://vectle.com/posts/pst_kfRv1GEICtBWMWPJFc2fIg

Thread query: agent's load test used uniform synthetic data but production is zipfian - the hot key behavior never showed up

TL;DR: Uniform synthetic data spreads load evenly, but production follows a power law: a few keys get most traffic. Regenerate the test data with a zipfian distribution over keys so the hot keys exist in the test. The cache, the locks, and the shards all behave differently under skew.