v0.24 Upgrade causing disk space issues on servers

v0.24 Upgrade causing disk space issues on servers

What was going on Issue typesense/typesense#928 (closed, 15 comments): RocksDB garbage collection not clearing disk up after collections marked for deletion. This is causing the disk space on the VMs our typesense servers are running on to run out of disk within a couple of days (150gb). Image 1 below shows the data folder slowly bloating - with our /data/db directory being clogged full of .sst files (Image 2). img1 [image] img2 [image] Upgrade servers to new v0.24. Run a scheduled job to write to our typesense servers - the job being a bulk write to the servers and allow for the job to run several times, causing a build up of non deleted redudant files.

The fix If disk space balloons after upgrading to Typesense 0.24 (RocksDB not compacting after collection drops), upgrade to 0.25.x. The maintainer added automatic compaction on collection drop, and the reporter confirmed their db directory shrank from 23G to 1.5G after dropping collections. Note: 0.25 upgrades RocksDB, so you cannot downgrade back to 0.24 on the same data directory.

In gh:typesense/typesense#928 the issue was: Issue typesense/typesense#928 (closed, 15 comments): RocksDB garbage collection not clearing disk up after collections marked for deletion. This is causing the disk space on the VMs our typesense servers are running on to run out of disk within a couple of days (150gb). Image 1 below shows the data folder slowly bloating - with our /data/db directory being clogged full of .sst files (Image 2). img1 [image] img2 [image] Upgrade servers to new v0.24. Run a scheduled job to write to our typesense servers - the job being a bulk write to the servers and allow for the job to run several times, causing a build up of non deleted redudant files. The verified answer: If disk space balloons after upgrading to Typesense 0.24 (RocksDB not compacting after collection drops), upgrade to 0.25.x. The maintainer added automatic compaction on collection drop, and the reporter confirmed their db directory shrank from 23G to 1.5G after dropping collections. Note: 0.25 upgrades RocksDB, so you cannot downgrade back to 0.24 on the same data directory. Source: https://github.com/typesense/typesense/issues/928