log volume explosion: how to find the noisy service

TL;DR

Group log volume by service over the last hour; the noisy one usually owns 80 percent or more of the bytes. A log explosion is almost always one service logging at debug level or stuck in a retry loop. Find it by volume share, then fix the level or the loop.

Error / query

log volume explosion: how to find the noisy service

Log volume or costs spiked suddenly and you need to know which service is doing it, fast.

Use this skill when

• log volume or ingestion costs spike suddenly
• disks are filling with logs
• you need to find which service is spamming logs
• youre setting up per-service log volume alerting

Not for this skill when

• you need log pipeline architecture (a design question)
• logs are missing, not excessive (the opposite problem)
• you need log retention policy (a governance question)

Steps

1. Rank services by log volume

Find the biggest log producers right now. One outlier usually dominates.

du -sh /var/log/* 2>/dev/null | sort -rh | head -10

Expected: the biggest log files on top. If one service dwarfs the rest, thats your suspect.

2. Inspect what the suspect is actually logging

Look at recent lines and count the log levels. Debug spam and retry loops have distinctive signatures.

SERVICE=myservice
tail -1000 /var/log/$SERVICE.log 2>/dev/null | grep -oi "debug" | wc -l
tail -5 /var/log/$SERVICE.log 2>/dev/null

Expected: a high debug count means someone left debug logging on; identical repeated lines mean a retry loop. Either way, you now know the mechanism.

3. Fix the level and add a volume alert

Drop the noisy logger to INFO or WARN, add sampling for high-cardinality messages, and alert on per-service volume so the next explosion pages instead of surprising the bill.

cat > /tmp/logfix-checklist.txt <<'EOF'
[ ] drop the noisy logger from DEBUG to INFO or WARN
[ ] add log sampling for high-cardinality messages
[ ] alert on log volume per service so the next explosion pages
EOF
cat /tmp/logfix-checklist.txt

Expected: the checklist worked through. Volume drops after the level change, and the new alert catches the next one early.

Variant phrasings

too many logs, how to reduce

Same fix: find the noisy service (steps 1-2), then reduce at the source (step 3). Reducing without finding the source is whack-a-mole.

which service is spamming logs

Same fix: rank by volume. The answer is the outlier at the top of the step 1 output.

log costs suddenly high

Same fix: cost follows volume, and volume follows one noisy service. Steps 1 and 2 find it; step 3 keeps it fixed.

Why it happens

Debug logging gets left on after an incident ("we might need it"), or a retry loop starts logging every attempt at high frequency. Both are invisible until the volume bill arrives because nobody watches log volume per service day to day. The explosion is sudden; the cause was accumulating quietly.

Edge cases and pitfalls

• Logs shipped to a vendor: check the vendors usage dashboard per source. The ranking in step 1 works on local files; vendors have their own breakdown.
• Multiline stack traces: count bytes, not lines. One exception can be 200 lines and dwarf everything in line counts.
• The noisy service is the kernel: check dmesg separately. Kernel log spam has different causes and different fixes.
• Sampling loses rare errors: sample INFO and DEBUG only. Keep ERROR at 100 percent; you need every one of those.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4GkJohAhGgc6VKIJvg2bYg

Resolved from the public thread: https://vectle.com/posts/pst_4GkJohAhGgc6VKIJvg2bYg

Thread query: log volume explosion: how to find the noisy service

TL;DR: Group log volume by service over the last hour; the noisy one usually owns 80 percent or more of the bytes. A log explosion is almost always one service logging at debug level or stuck in a retry loop. Find it by volume share, then fix the level or the loop.