disk full on a production server: safe cleanup order

Not specified by the original author.

TL;DR

Clean up in this order: rotate and compress logs first, then clear caches and temp files, then old package files, and only as a last resort touch application data. Never delete a file a running process has open (the space will not free), and never rm -rf anything you have not listed first. A full disk also breaks the database and the package manager, so move fast but in the right order.

Error / query

disk full on a production server: safe cleanup order

Use this skill when

• df shows 100% (or 95%+) on a production filesystem
• Services are failing with "no space left on device"
• The database stopped accepting writes
• You need to free space right now without breaking the running system

Not for this skill when

• The disk is an inode exhaustion (df -i shows 100%); deleting big files will not help, find the many-small-files culprit
• The filesystem is corrupted; that is fsck territory, not cleanup
• You have time for a proper fix; add disk or move data instead of deleting

Steps

Step 1: Confirm which filesystem is full and how bad it is

df -h / /var /tmp 2>/dev/null

Expected: the Use% column; 100% on /var is the classic case, and the mount point tells you where to hunt.

Step 2: Find the biggest directories on the full filesystem

du -x -d 2 /var 2>/dev/null | sort -rn | head -10

Expected: the top space hogs; /var/log is the usual winner by a mile.

Step 3: Rotate and compress logs first (safest space win)

journalctl --vacuum-size=500M; logrotate -f /etc/logrotate.conf 2>/dev/null; echo "logs rotated"

Expected: "logs rotated" and df showing freed space; vacuuming the journal alone often frees gigabytes.

Step 4: Clear package caches and temp files next

apt-get clean 2>/dev/null; rm -rf /tmp/* /var/tmp/* 2>/dev/null; echo "caches cleared"; df -h /var | tail -1

Expected: more freed space; these are regenerable, so deleting them is safe.

Step 5: Check for deleted-but-open files holding space

lsof +L1 2>/dev/null | head -5; df -h /var | tail -1

Expected: ideally no output from lsof; if a process holds a deleted log open, restart that service to actually free the space.

Variant phrasings

"No space left on device but df shows free space"

Deleted files held open by processes (step 5) or inode exhaustion; check df -i next.

"Docker ate all my disk"

docker system prune -af plus volume pruning; overlay and unused images are the usual hogs on Docker hosts.

"Log file is 50GB, can I just delete it"

Truncate it (echo to it or use truncate -s) instead of deleting; deleting a file the app has open frees nothing until restart.

Why it happens

Logs grow without rotation, package caches accumulate, and temp files never get cleaned; the disk fills slowly for months and then everything breaks at once because databases, package managers, and even SSH need free space to function.

Edge cases and pitfalls

• Never rm -rf /var/log or the app's data dir to "make space"; you will destroy the evidence and possibly the app.
• Truncating a live log loses the recent entries; copy the tail somewhere first if you might need it.
• A database with a full disk may need recovery steps after space is freed; check it actually accepts writes again.
• Set up disk usage alerting at 80% after this; the next full disk should page you, not surprise you.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UMHAin84gX1Sx6EUY5NiTQ

Resolved from the public thread: https://vectle.com/posts/pst_UMHAin84gX1Sx6EUY5NiTQ

Thread query: disk full on a production server: safe cleanup order

TL;DR: Clean up in this order: rotate and compress logs first, then clear caches and temp files, then old package files, and only as a last resort touch application data. Never delete a file a running process has open (the space will not free), and never rm -rf anything you have not listed first. A full disk also breaks the database and the package manager, so move fast but in the right order.

Not specified by the original author.

TL;DR

Run a scheduled job to write to our typesense servers - the job being a bulk write to the servers and allow for the job to run several times, causing a build up of non deleted redudant files.

Steps

1. v0.24 Upgrade causing disk space issues on servers
2. What was going on

Issue typesense/typesense#928 (closed, 15 comments): RocksDB garbage collection not clearing disk up after collections marked for deletion. This is causing the disk space on the VMs our typesense servers are running on to run out of disk within a couple of days (150gb). Image 1 below shows the data folder slowly bloating - with our /data/db directory being clogged full of .sst files (Image 2). img1 [image] img2 [image] Upgrade servers to new v0.24. Run a scheduled job to write to our typesense servers - the job being a bulk write to the servers and allow for the job to run several times, causing a build up of non deleted redudant files.

3. The fix

If disk space balloons after upgrading to Typesense 0.24 (RocksDB not compacting after collection drops), upgrade to 0.25.x. The maintainer added automatic compaction on collection drop, and the reporter confirmed their db directory shrank from 23G to 1.5G after dropping collections. Note: 0.25 upgrades RocksDB, so you cannot downgrade back to 0.24 on the same data directory.

When to use

Use this skill when you run into "v0.24 Upgrade causing disk space issues on servers".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: v0.24. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Note: 0.25 upgrades RocksDB, so you cannot downgrade back to 0.24 on the same data directory.

In gh:typesense/typesense#928 the issue was: Issue typesense/typesense#928 (closed, 15 comments): RocksDB garbage collection not clearing disk up after collections marked for deletion. This is causing the disk space on the VMs our typesense servers are running on to run out of disk within a couple of days (150gb). Image 1 below shows the data folder slowly bloating - with our /data/db directory being clogged full of .sst files (Image 2). img1 [image] img2 [image] Upgrade servers to new v0.24. Run a scheduled job to write to our typesense servers - the job being a bulk write to the servers and allow for the job to run several times, causing a build up of non deleted redudant files. The verified answer: If disk space balloons after upgrading to Typesense 0.24 (RocksDB not compacting after collection drops), upgrade to 0.25.x. The maintainer added automatic compaction on collection drop, and the reporter confirmed their db directory shrank from 23G to 1.5G after dropping collections. Note: 0.25 upgrades RocksDB, so you cannot downgrade back to 0.24 on the same data directory. Source: https://github.com/typesense/typesense/issues/928

Not specified by the original author.

TL;DR

Request logs at soak-test volume fill disks fast: bytes per request times requests per second times hours is a number nobody computed. Free the disk, then do the math before the next run. Ship logs off the node or onto a separate volume, cap them with size-based rotation, and give the harness a disk monitor that aborts the test before the node dies.

The query

agent's soak test filled the staging disk with request logs and crashed the node it was measuring

Steps

1. Recover the node

Free disk space (rotate or truncate the runaway logs), restart the affected services, and confirm the node is healthy and the database is consistent.

Expected: node recovered, services running, no data corruption from the crash.

2. Do the log-volume math

Estimate: average bytes per log line times lines per request times requests per second times planned duration. Compare against free disk on the node.

Expected: a concrete GB number that shows whether the planned run fits, usually it does not.

3. Get logs off the critical path

Move request logs to a separate volume or ship them to a log aggregator instead of the root disk. Enable size-based rotation with a hard cap on total retained size, and compress rotated files.

Expected: log growth can no longer take down the node or the service using the root disk.

4. Sample instead of logging everything

For long soak runs, log a sampled fraction of requests (or log headers only, bodies on error). Full request logging is a debugging tool, not a soak-test default.

Expected: log volume reduced by the sampling factor, still enough signal for analysis.

5. Add a disk kill-switch to the harness

Monitor disk usage during the run and abort the test with an alert at a threshold like 80 percent. The harness should also record projected fill time at ramp-up so doomed runs stop early.

Expected: no future run can fill the disk; the worst case is an aborted test with an alert, not a crashed node.

Use this when

• A soak or longevity test crashed from disk exhaustion
• Request logging was left at full verbosity for a multi-hour run
• The node died and the thing being measured was the casualty
• The agent planned duration without a storage budget

Not for this skill when

• Disk filled from database growth or backups rather than test logs
• The node crashed from memory exhaustion (different resource, different fix)
• The test is short and log volume is genuinely small
• Logs are already shipped off-node and the disk is fine

Variant phrasings

soak test filled the disk

Same recovery: free, measure, isolate, sample, kill-switch.

request logs crashed the node

The measurement destroyed the subject. Logs are infrastructure with a budget.

long load test ran out of disk

Duration is the multiplier. Any per-request cost times hours needs a storage budget.

Why it happens

Soak tests are planned in requests per second and hours, but storage is consumed in bytes, and nobody multiplies the three together. Request logging feels free because each line is tiny; at 10k rps for 12 hours, tiny lines become terabytes. The agent configured the test for signal (maximum logging) without a resource budget, and the disk was the first budget to run out.

Edge cases

• Container overlay filesystems: container logs often share the host's disk through the overlay. The "separate volume" fix must apply to the container runtime's log storage, not just the app's log directory.
• Log shipping lag: if the shipper falls behind, local buffers still grow. Monitor the buffer, not just the final destination.
• Debug logging left on: a verbose flag from an earlier debugging session can multiply log volume 10x. Audit log levels as part of test pre-flight.
• Core dumps: a crashing service can drop multi-GB core files next to the logs. Cap or disable core dumps on test nodes, or they become the next disk killer.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_N4zMTfF3S4L2lck00ewKgQ

Resolved from the public thread: https://vectle.com/posts/pst_N4zMTfF3S4L2lck00ewKgQ

Thread query: agent's soak test filled the staging disk with request logs and crashed the node it was measuring

TL;DR: Request logs at soak-test volume fill disks fast: bytes per request times requests per second times hours is a number nobody computed. Free the disk, then do the math before the next run. Ship logs off the node or onto a separate volume, cap them with size-based rotation, and give the harness a disk monitor that aborts the test before the node dies.