There Are A Lot Of Single SSTable Compactions That Don’t Compact Anything¶

Phenomena¶

ScyllaDB’s CPU utilization is unexpectedly high and we see too many compactions while there are not many WRITE/UPDATE/DELETE/TTLed operations and you see many messages like this in the syslog:

compaction - Compacted 1 SSTables to ... (~100% of original) ...

Problem¶

ScyllaDB SSTables can have expired or soon to be expired tombstones in them and there is a need to clean them up eventually. Tombstones can be generated by DELETE operations, TTL data, insertion of null fields or usage of collections. Compactions will get rid of expired tombstones, but if there are no compactions currently happening, ScyllaDB may apply heuristics to force compactions on a lone table that has a certain ratio of expired tombstones.

To validate that the SSTable being compacted indeed has tombstones:

sstablemetadata /var/lib/scylla/data/some_ks/some_cf-UUID/some_ks-ka-*-Data.db | grep "Estimated droppable tombstones:"

The output may be something like this:

Estimated droppable tombstones: 0.08672144669324656
Estimated droppable tombstones: 0.08935147496746765
Estimated droppable tombstones: 0.11850671228977766
Estimated droppable tombstones: 0.08511697234511045
Estimated droppable tombstones: 0.035764466964587543
Estimated droppable tombstones: 0.1894537114259597
Estimated droppable tombstones: 0.13213090183839682
Estimated droppable tombstones: 0.1208004932191389
Estimated droppable tombstones: 0.07019800509981686
Estimated droppable tombstones: 0.12196540423823726
Estimated droppable tombstones: 0.11981627152867445
Estimated droppable tombstones: 0.11780386543765468
Estimated droppable tombstones: 0.17214256718348064

Heuristics is not a perfect solution and it may issue a compaction in cases where there are no expired tombstones to drop. Should this happen, a tombstone compaction may begin needlessly, wasting CPU and I/O resources resulting in a compacted table which shows no change from the original SSTable.

Solution¶

1. Prevent automatic tombstones compactions by increasing the value of tombstone_compaction_interval to some big value (it’s integer value representing a time period in seconds).

For example this will change it to 4 days:

ALTER table <your table name> with compaction = { 'class' : 'SizeTieredCompactionStrategy', 'tombstone_compaction_interval' : 345600 };

2. Make sure unchecked_tombstone_compaction is set to false. If it is set to true, tombstone_threshold will be ignored, and compaction will be run whenever tombstone_compaction_interval has elapsed.

ALTER table <your table name> with compaction = { 'class' : 'SizeTieredCompactionStrategy', 'unchecked_tombstone_compaction' : false };

Troubleshoot