## TL;DR
Container logs are just files on the node, and without rotation they grow until the disk fills. Turn on the kubelet's container log rotation (`--container-log-max-size` and `--container-log-max-files`), which caps each container's log files and keeps a few rotations. Then find the noisiest containers and fix their log volume, because rotation caps the damage but the app is still writing too much.

## Error / query
```text
container log rotation: keep node disk from filling
```

## Use this skill when
- Nodes report DiskPressure and `/var/log/containers` is huge
- `du` shows container log JSON files in the gigabytes
- A verbose app floods logs and threatens node stability
- You want log rotation as a default on every node

## Not for this skill when
- Disk is full from images or volumes (different cleanup)
- You need the log content shipped somewhere (logging pipeline)
- A single runaway process inside a pod writes to a mounted volume

## Steps

### Step 1: Confirm logs are the disk hog
```bash
du -sh /var/log/containers /var/log/pods 2>/dev/null
du -sh /var/log/pods/*/* 2>/dev/null | sort -rh | head -5
```
Expected: the top directories point at specific pods. If containers/pods dominate disk usage, log rotation is the fix; if not, look at images or volumes.

### Step 2: Enable kubelet container log rotation
```bash
# in the kubelet config file or systemd drop-in:
containerLogMaxSize: "50Mi"
containerLogMaxFiles: 5
```
Expected: after a kubelet restart, each container keeps at most 5 files of 50Mi each. The kubelet rotates automatically; no cron job needed. On managed clusters set this through the node group's kubelet config, not by SSH.

### Step 3: Restart the kubelet and verify rotation
```bash
systemctl restart kubelet
ls -la /var/log/pods/[namespace]_[pod]_[uid]/[container]/ | head
```
Expected: you see numbered rotated files (`0.log`, `1.log.gz` etc. depending on runtime) instead of one ever-growing file. New writes go to the current file.

### Step 4: Find and quiet the noisiest loggers
```bash
du -sh /var/log/pods/*/* 2>/dev/null | sort -rh | head -10
```
Expected: the same few pods at the top after rotation is on. Those apps need their log level raised (info to warn), request logging sampled, or health-check endpoints excluded from access logs. Rotation is the guardrail; fixing volume is the cure.

## Variant phrasings

### "kubernetes container logs filling disk"
Steps 1 and 2. Rotation caps it today; step 4 fixes the source.

### "kubelet log rotation config"
`containerLogMaxSize` and `containerLogMaxFiles` in the kubelet config, step 2.

## Why it happens
The container runtime writes every line the app prints to a JSON log file on the node, and by default nothing rotates or deletes those files. One debug-level app can write gigabytes a day. The kubelet only starts evicting pods at DiskPressure thresholds, by which time the node is already unhealthy.

## Edge cases and pitfalls
- Rotation settings apply per container, so 50Mi x 5 files x many containers still adds up; size for your densest node.
- `kubectl logs` only shows the current plus `--previous` container logs; rotated files are node-local and need direct access.
- Changing log paths or drivers (journald, etc.) changes where rotation applies; check your runtime config.
- If logs must be retained for compliance, ship them to centralized storage first; rotation deletes local history.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O3jJnUzPbH3fpi7aDuRd3Q