scylla 'compaction' falling behind under write load: how to tell

TL;DR

Compaction falls behind when memtable flushes create SSTables faster than compaction can merge them. Confirm it the honest way: sample nodetool compactionstats twice and watch pending tasks grow, then check which table owns the SSTable pileup. The fix is on the write side: bigger batches, fewer flushes, and only then more compaction throughput. Read latency recovers as the backlog drains.

nodetool compactionstats: pending tasks: 1,284 and growing; nodetool tablestats: SSTable count 412 on keyspace.events

Steps

1. Run nodetool compactionstats, wait five minutes, and run it again. Expected: pending tasks are growing, not draining. A one-time spike drains; a growing number means compaction has lost the race.

2. Run nodetool tablestats and look at the SSTable count per table. Expected: one table holds hundreds of SSTables while the rest are quiet. That is the table outrunning compaction.

3. Find why flushes are so frequent. Check the log for memtable flush messages on that table and look at your write pattern: many tiny writes flush small memtables constantly. Expected: flushes every few seconds on the hot table, each producing a small SSTable.

4. Reduce flush frequency. Increase the memtable size settings or batch writes client-side so each flush carries more data. Expected: fewer, larger SSTables per hour, and pending compactions start draining.

5. Raise compaction throughput. Increase compactionthroughputmbpersec in scylla.yaml, keeping an eye on disk IO so compaction does not starve reads. Expected: the pending-task backlog drains faster.

6. Check for tombstone overload. Heavy delete workloads create compaction work that never shrinks the data. Expected: if deletes dominate, tune TTLs or expiry so the deletes stop generating endless compaction in the first place.

Use this when

• Pending compactions grow over time on a write-heavy cluster.
• One table shows hundreds of SSTables in tablestats.
• Read latency climbs while write throughput looks normal.
• Disk usage grows faster than the actual data volume.

Not for this skill when

• Compaction is idle and pending tasks sit at zero. Your latency problem is elsewhere.
• A single compaction task is stuck for hours on one huge SSTable. That is a stuck compaction, a different problem.
• The node just bootstrapped or repaired. Backlogs after topology changes drain on their own.

Variant phrasings

• scylladb compaction pending tasks growing
• scylla too many sstables read slow
• scylladb compactionstats pending never drains
• scylla memtable flush too frequent
• scylladb tombstone compaction overload

Why it happens

ScyllaDB buffers writes in memtables and flushes them to SSTables on disk; a background compaction process continuously merges SSTables so reads stay fast. When writes arrive as a firehose of tiny batches, flushes produce small SSTables faster than compaction merges them. Reads then have to consult hundreds of SSTables, latency climbs, and disk fills with overlapping data. The write path looks healthy while the merge path drowns.

Edge cases

• Cranking compaction throughput too high steals disk IO from reads and can make latency worse during the catch-up. Raise it gradually.
• Wide partitions compact slowly and can stall a whole compaction round behind one giant SSTable. Check for large-partition warnings alongside the backlog.
• Lowering the memtable flush threshold "to save memory" is a classic self-inflicted cause: it multiplies SSTable count. Do not shrink memtables on a write-heavy table.
• Major compactions rewrite everything and create huge temporary disk usage. Avoid them as a routine fix; fix the flush rate instead.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstL8nn_qo4KRji1igK-Dl8A

Resolved from the public thread: https://vectle.com/posts/pstL8nn_qo4KRji1igK-Dl8A

Thread query: scylla 'compaction' falling behind under write load: how to tell

TL;DR: Compaction falls behind when memtable flushes create SSTables faster than compaction can merge them. Confirm it the honest way: sample nodetool compactionstats twice and watch pending tasks grow, then check which table owns the SSTable pileup. The fix is on the write side: bigger batches, fewer flushes, and only then more compaction throughput. Read latency recovers as the backlog drains.