bigtable 'hot tablet' warnings on a sequential row key: how to salt the key

TL;DR

Salt the row key. Sequential keys like timestamps or auto-increment ids send every new write to the same tablet on the same node, which is exactly what a hot tablet is. Prepend a hash of the natural key modulo N, or reverse the timestamp for time-ordered data, and the writes spread out. Confirm the hotspot in Key Visualizer before you redesign, and again after.

Key Visualizer: one tablet range glowing bright (nearly all write traffic) while the rest of the instance is dark; per-node CPU shows one node at 90%+ with the others near idle.

Steps

1. Open Key Visualizer for the instance and look at the write heatmap over the affected time window. Expected: a single narrow key range glowing bright while the rest is dark. That is the hot tablet.

2. Check per-node CPU in Cloud Monitoring for the same window. Expected: one node pegged near max while the others idle. Imbalance confirms a hotspot rather than just fast growth.

3. Redesign the row key. Prepend a salt bucket, for example hash of the natural key mod 8 followed by a separator and the original key, so writes fan out across 8 buckets. For purely time-ordered data, a reversed timestamp works too. Expected: new writes spread across tablets instead of piling onto one.

4. If you still need range scans on the original key, keep the bucket count small and fixed, and fan scans out across the N bucket prefixes. Expected: scans touch N ranges instead of one, which is the price of the salt, and it stays manageable.

5. Migrate by writing new data in the salted schema and backfilling; old hot tablets split once traffic spreads. Expected: Key Visualizer evens out and per-node CPU converges within a day of the cutover.

Use this when

• Bigtable flags a hot tablet, or you see one node hot and the rest idle.
• Row keys are sequential: timestamps, increasing ids, or date-prefixed keys.
• Write latency spikes correlate with traffic peaks on one node.
• Key Visualizer shows a bright narrow band on writes.

Not for this skill when

• All nodes are hot together. That is under-provisioning, add nodes instead.
• The hotspot is on reads of a few famous rows. That needs caching or read replicas of the pattern, not key salting.
• You are on the emulator. Hotspotting behavior there does not reflect production.

Variant phrasings

• bigtable hotspotting sequential row keys
• cloud bigtable one node high CPU others idle
• bigtable key visualizer hot tablet
• how to design bigtable row key to avoid hotspots
• bigtable reversed timestamp row key

Why it happens

Bigtable splits tables into tablets by contiguous row-key ranges, and each tablet is served by one node. Sequential keys mean every new write lands at the "end" of the key space, which is one tablet on one node, so that node absorbs the entire write load while the others do nothing. Salting breaks the correlation between insertion order and key order, spreading fresh writes across many tablets.

Edge cases

• Too many salt buckets makes range scans expensive, since each scan fans out across all buckets. Start small, like 8 to 16.
• Reversed timestamps fix write hotspots but destroy chronological range scans. Only use them when you rarely scan by time range.
• Field promotion (putting a high-cardinality field first in the key) is often cleaner than a hash salt when you have one.
• Salting changes the key, so every reader must know the scheme. Document it next to the table definition or someone will query the raw key and find nothing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstuMhus-7-DbhyElG8ky6Gw

Resolved from the public thread: https://vectle.com/posts/pstuMhus-7-DbhyElG8ky6Gw

Thread query: bigtable 'hot tablet' warnings on a sequential row key: how to salt the key

TL;DR: Salt the row key. Sequential keys like timestamps or auto-increment ids send every new write to the same tablet on the same node, which is exactly what a hot tablet is. Prepend a hash of the natural key modulo N, or reverse the timestamp for time-ordered data, and the writes spread out. Confirm the hotspot in Key Visualizer before you redesign, and again after.