container log rotation: keep node disk from filling

TL;DR

Container logs are just files on the node, and without rotation they grow until the disk fills. Turn on the kubelet's container log rotation (--container-log-max-size and --container-log-max-files), which caps each container's log files and keeps a few rotations. Then find the noisiest containers and fix their log volume, because rotation caps the damage but the app is still writing too much.

Error / query

container log rotation: keep node disk from filling

Use this skill when

• Nodes report DiskPressure and /var/log/containers is huge
• du shows container log JSON files in the gigabytes
• A verbose app floods logs and threatens node stability
• You want log rotation as a default on every node

Not for this skill when

• Disk is full from images or volumes (different cleanup)
• You need the log content shipped somewhere (logging pipeline)
• A single runaway process inside a pod writes to a mounted volume

Steps

Step 1: Confirm logs are the disk hog

du -sh /var/log/containers /var/log/pods 2>/dev/null
du -sh /var/log/pods/*/* 2>/dev/null | sort -rh | head -5

Expected: the top directories point at specific pods. If containers/pods dominate disk usage, log rotation is the fix; if not, look at images or volumes.

Step 2: Enable kubelet container log rotation

# in the kubelet config file or systemd drop-in:
containerLogMaxSize: "50Mi"
containerLogMaxFiles: 5

Expected: after a kubelet restart, each container keeps at most 5 files of 50Mi each. The kubelet rotates automatically; no cron job needed. On managed clusters set this through the node group's kubelet config, not by SSH.

Step 3: Restart the kubelet and verify rotation

systemctl restart kubelet
ls -la /var/log/pods/[namespace]_[pod]_[uid]/[container]/ | head

Expected: you see numbered rotated files (0.log, 1.log.gz etc. depending on runtime) instead of one ever-growing file. New writes go to the current file.

Step 4: Find and quiet the noisiest loggers

du -sh /var/log/pods/*/* 2>/dev/null | sort -rh | head -10

Expected: the same few pods at the top after rotation is on. Those apps need their log level raised (info to warn), request logging sampled, or health-check endpoints excluded from access logs. Rotation is the guardrail; fixing volume is the cure.

Variant phrasings

"kubernetes container logs filling disk"

Steps 1 and 2. Rotation caps it today; step 4 fixes the source.

"kubelet log rotation config"

containerLogMaxSize and containerLogMaxFiles in the kubelet config, step 2.

Why it happens

The container runtime writes every line the app prints to a JSON log file on the node, and by default nothing rotates or deletes those files. One debug-level app can write gigabytes a day. The kubelet only starts evicting pods at DiskPressure thresholds, by which time the node is already unhealthy.

Edge cases and pitfalls

• Rotation settings apply per container, so 50Mi x 5 files x many containers still adds up; size for your densest node.
• kubectl logs only shows the current plus --previous container logs; rotated files are node-local and need direct access.
• Changing log paths or drivers (journald, etc.) changes where rotation applies; check your runtime config.
• If logs must be retained for compliance, ship them to centralized storage first; rotation deletes local history.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O3jJnUzPbH3fpi7aDuRd3Q