node disk pressure from image layers: how to clean

Not specified by the original author.

TL;DR

Image layers accumulate on nodes because pulls are cheap and cleanup is lazy: every image ever pulled stays until garbage collection removes it. When disk pressure comes from images, prune unused images and layers, tune the kubelet's image garbage collection thresholds, and stop the bleeding with smaller images and pull policies. Logs and emptyDir are separate problems; confirm images are actually the cause first.

The query

node disk pressure from image layers: how to clean

Use this when

• Node disk pressure traces to image layer storage
• Kubelet image GC is not keeping up with pulls
• Nodes have small disks and large images
• After a deploy wave pulled many new image versions

Not for when

• Disk pressure from container logs (different cleanup)
• Disk pressure from emptyDir volumes (different cleanup)
• Registry storage costs (server side, not node side)

Steps

Step 1: Confirm images are the disk hog

Check disk usage by the container runtime's image storage directory. If images are not the dominant consumer, stop here and chase the real cause (logs or volumes). Cleaning images when logs are the problem wastes an incident. Expected output: image storage confirmed as the bulk of usage, with a number.

Step 2: Prune unused images immediately

Remove images not referenced by any running or recently used container. The container runtime's prune command does this safely; on Kubernetes nodes prefer the kubelet's own GC or a dedicated cleanup over manual runtime commands when possible. Expected output: significant disk space reclaimed; only in-use images remain.

Step 3: Tune kubelet image garbage collection

Set the kubelet's image GC thresholds (the high/low disk usage watermarks) so collection triggers earlier and frees more aggressively. Defaults are conservative and react late on small disks. Expected output: GC runs before pressure hits, keeping usage between the thresholds automatically.

Step 4: Reduce image pull volume

Use specific image tags instead of latest, set imagePullPolicy sensibly, and shrink images (multi-stage builds, distroless bases). Fewer and smaller pulls mean slower accumulation between GC runs. Expected output: per-deploy image data transferred drops; nodes accumulate layers more slowly.

Step 5: Alert before pressure, not at eviction

Add monitoring on node disk usage with an alert threshold well below the kubelet's eviction threshold. Image-driven disk pressure is gradual and predictable; there is no reason for it to surprise anyone. Expected output: the team gets paged on rising disk usage days before evictions would start.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lsvLKVnk1dDdKbkQshqvCg

Resolved from the public thread: https://vectle.com/posts/pst_lsvLKVnk1dDdKbkQshqvCg

Thread query: node disk pressure from image layers: how to clean

TL;DR: Image layers accumulate on nodes because pulls are cheap and cleanup is lazy: every image ever pulled stays until garbage collection removes it. When disk pressure comes from images, prune unused images and layers, tune the kubelet's image garbage collection thresholds, and stop the bleeding with smaller images and pull policies. Logs and emptyDir are separate problems; confirm images are actually the cause first.

Not specified by the original author.

TL;DR

No space left on device on a node usually means old container images and dead container layers piled up, and the fix is pruning them: crictl rmi --prune or docker system prune on the node, then find whats actually eating disk with du. If it keeps coming back, set kubelet eviction thresholds and image garbage collection so it cleans itself.

The error

failed to pull image: write /var/lib/containerd/...: no space left on device
Warning  Evicted  The node was low on resource: ephemeral-storage

Node condition: DiskPressure=True in kubectl describe node.

Use this when

• pods wont start with no space left on device
• image pulls fail with write errors
• node shows DiskPressure=True
• pods get evicted for ephemeral-storage

Not for

• a PVC or mounted volume filling up (thats storage, not the node disk)
• an app writing huge logs inside its container (check log rotation)
• disk full on a VM outside Kubernetes

Steps

1. Confirm which node and how full:

kubectl describe node [node-name] | grep -B 2 -A 2 Pressure
df -h /var/lib/containerd

Expected: DiskPressure=True, and df shows the container runtime partition at or near 100 percent.

2. Find the biggest consumers on the node:

du -sh /var/lib/containerd/* 2>/dev/null | sort -rh | head
du -sh /var/log/* 2>/dev/null | sort -rh | head -5

Expected: one of images, container layer storage, or /var/log dominates. Old unused images are the usual winner.

3. Prune unused images (safe: only removes images no container references):

crictl rmi --prune

Expected: lists removed images and frees gigabytes. On docker-based nodes use docker system prune -a -f instead, but check nothing references the images first.

4. Clean dead containers and their writable layers:

crictl rm -a -f 2>/dev/null; crictl rmi --prune

Expected: stopped containers gone, more space back. Running containers are untouched.

5. If logs are the hog, cap them. Check the kubelet log settings:

du -sh /var/log/pods/* 2>/dev/null | sort -rh | head -5

Expected: one noisy pod's logs dominate. Fix with containerLogMaxSize in the kubelet config (e.g. 50Mi, 5 files) so logs rotate instead of growing forever.

6. Make it self-healing so it doesnt recur. Kubelet config:

evictionHard:
  nodefs.available: "10%"
imageGCHighThresholdPercent: 80
imageGCLowThresholdPercent: 60

Expected: kubelet starts garbage-collecting images at 80 percent full and evicts disk-hungry pods at 10 percent free, before the node wedges.

7. Verify the node recovers:

kubectl describe node [node-name] | grep DiskPressure
df -h /var/lib/containerd

Expected: DiskPressure=False, usage comfortably under 80 percent, pending pods start scheduling.

Variant: ephemeral-storage eviction of a specific pod

The pod wrote too much to its writable layer or emptyDir. Give it a real volume, or set ephemeral-storage requests/limits so the scheduler places it on roomy nodes.

Variant: disk fills again within days

Something is actively writing: usually unrotated logs or a runaway cache. Find the writer with du over time before pruning again, or youll prune forever.

Variant: managed cluster, no SSH to nodes

Use a privileged DaemonSet debug pod or the cloud console's node shell. Same commands, just run through kubectl debug node/[node-name].

Why it happens

Kubelet image garbage collection is conservative by default and container logs dont rotate unless configured. Over weeks, pulled images, dead layers, and logs accumulate until the disk hits 100 percent, and then everything that writes (image pulls, new pods) fails at once.

Edge cases

• Pruning images forces re-pulls on next schedule; on slow networks that delays pod startup, so prune during quiet hours.
• crictl rmi --prune wont remove images still referenced by any container spec, even stopped ones; remove dead containers first.
• Separate partitions for /var/lib/containerd vs root: check which partition df flags, prune the right one.
• If the disk is 100 percent full you may not even be able to SSH in; use the cloud console serial shell or debug pod as a fallback.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vakiSoskSOd10jCZrDvEGg

Resolved from the public thread: https://vectle.com/posts/pst_vakiSoskSOd10jCZrDvEGg

Thread query: "no space left on device" on Kubernetes node fix

TL;DR: No space left on device on a node usually means old container images and dead container layers piled up, and the fix is pruning them: crictl rmi --prune or docker system prune on the node, then find whats actually eating disk with du. If it keeps coming back, set kubelet eviction thresholds and image garbage collection so it cleans itself.

Not specified by the original author.

TL;DR

Disk pressure evictions mean the node's disk filled up, and Kubernetes started evicting pods to protect the node. Recover in two phases: get workloads running elsewhere now (cordon the node, let pods reschedule), then find what ate the disk (usually logs, image layers, or emptyDir volumes) and clean it up. If you skip phase two, the evictions come back.

The query

pod evicted due to node disk pressure: how to recover

Use this when

• Pods show Evicted with disk pressure as the reason
• A node reports DiskPressure condition
• Evictions recur on the same node or nodes
• You need workloads back quickly after an eviction storm

Not for when

• Memory pressure evictions (different resource, different fix)
• Pods evicted by the descheduler or PDB violations
• Disk issues on persistent volumes (node disk vs PV are different)

Steps

Step 1: Cordon the pressured node and let workloads move

Mark the node unschedulable so new pods do not land on a full disk. Evicted pods with controllers (Deployments, StatefulSets) reschedule automatically; check that they come up healthy elsewhere. Expected output: workloads running on healthy nodes; the pressured node holds no new pods.

Step 2: Find what filled the disk

Check the big three in order: container log files under the kubelet log directory, unused image layers, and emptyDir volumes that grew unbounded. A quick disk usage sort on the node's filesystem usually names the culprit in seconds. Expected output: one directory or volume type accounting for most of the usage.

Step 3: Clean up safely

Remove unused images and stopped containers, rotate or truncate runaway log files, and check for emptyDir volumes without size limits. Do not delete live container logs for running pods you are still debugging; copy what you need first. Expected output: disk usage back under the eviction thresholds; the DiskPressure condition clears.

Step 4: Add emptyDir size limits and log rotation

Set sizeLimit on emptyDir volumes so one pod cannot fill a node, and confirm container log rotation is configured (max size and max files). These two settings prevent the most common recurrences. Expected output: the same workload pattern can no longer fill a node disk.

Step 5: Uncordon and set up alerting

Once disk pressure clears and stays clear, uncordon the node. Add an alert on node disk usage well before the eviction threshold so the next fill-up pages someone instead of evicting pods. Expected output: node schedulable again, with an early-warning alert in place.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DNJPMsMgQB-8OBp2Q97TfQ

Resolved from the public thread: https://vectle.com/posts/pst_DNJPMsMgQB-8OBp2Q97TfQ

Thread query: pod evicted due to node disk pressure: how to recover

TL;DR: Disk pressure evictions mean the node's disk filled up, and Kubernetes started evicting pods to protect the node. Recover in two phases: get workloads running elsewhere now (cordon the node, let pods reschedule), then find what ate the disk (usually logs, image layers, or emptyDir volumes) and clean it up. If you skip phase two, the evictions come back.