"no space left on device" on Kubernetes node fix

Not specified by the original author.

TL;DR

No space left on device on a node usually means old container images and dead container layers piled up, and the fix is pruning them: crictl rmi --prune or docker system prune on the node, then find whats actually eating disk with du. If it keeps coming back, set kubelet eviction thresholds and image garbage collection so it cleans itself.

The error

failed to pull image: write /var/lib/containerd/...: no space left on device
Warning  Evicted  The node was low on resource: ephemeral-storage

Node condition: DiskPressure=True in kubectl describe node.

Use this when

• pods wont start with no space left on device
• image pulls fail with write errors
• node shows DiskPressure=True
• pods get evicted for ephemeral-storage

Not for

• a PVC or mounted volume filling up (thats storage, not the node disk)
• an app writing huge logs inside its container (check log rotation)
• disk full on a VM outside Kubernetes

Steps

1. Confirm which node and how full:

kubectl describe node [node-name] | grep -B 2 -A 2 Pressure
df -h /var/lib/containerd

Expected: DiskPressure=True, and df shows the container runtime partition at or near 100 percent.

2. Find the biggest consumers on the node:

du -sh /var/lib/containerd/* 2>/dev/null | sort -rh | head
du -sh /var/log/* 2>/dev/null | sort -rh | head -5

Expected: one of images, container layer storage, or /var/log dominates. Old unused images are the usual winner.

3. Prune unused images (safe: only removes images no container references):

crictl rmi --prune

Expected: lists removed images and frees gigabytes. On docker-based nodes use docker system prune -a -f instead, but check nothing references the images first.

4. Clean dead containers and their writable layers:

crictl rm -a -f 2>/dev/null; crictl rmi --prune

Expected: stopped containers gone, more space back. Running containers are untouched.

5. If logs are the hog, cap them. Check the kubelet log settings:

du -sh /var/log/pods/* 2>/dev/null | sort -rh | head -5

Expected: one noisy pod's logs dominate. Fix with containerLogMaxSize in the kubelet config (e.g. 50Mi, 5 files) so logs rotate instead of growing forever.

6. Make it self-healing so it doesnt recur. Kubelet config:

evictionHard:
  nodefs.available: "10%"
imageGCHighThresholdPercent: 80
imageGCLowThresholdPercent: 60

Expected: kubelet starts garbage-collecting images at 80 percent full and evicts disk-hungry pods at 10 percent free, before the node wedges.

7. Verify the node recovers:

kubectl describe node [node-name] | grep DiskPressure
df -h /var/lib/containerd

Expected: DiskPressure=False, usage comfortably under 80 percent, pending pods start scheduling.

Variant: ephemeral-storage eviction of a specific pod

The pod wrote too much to its writable layer or emptyDir. Give it a real volume, or set ephemeral-storage requests/limits so the scheduler places it on roomy nodes.

Variant: disk fills again within days

Something is actively writing: usually unrotated logs or a runaway cache. Find the writer with du over time before pruning again, or youll prune forever.

Variant: managed cluster, no SSH to nodes

Use a privileged DaemonSet debug pod or the cloud console's node shell. Same commands, just run through kubectl debug node/[node-name].

Why it happens

Kubelet image garbage collection is conservative by default and container logs dont rotate unless configured. Over weeks, pulled images, dead layers, and logs accumulate until the disk hits 100 percent, and then everything that writes (image pulls, new pods) fails at once.

Edge cases

• Pruning images forces re-pulls on next schedule; on slow networks that delays pod startup, so prune during quiet hours.
• crictl rmi --prune wont remove images still referenced by any container spec, even stopped ones; remove dead containers first.
• Separate partitions for /var/lib/containerd vs root: check which partition df flags, prune the right one.
• If the disk is 100 percent full you may not even be able to SSH in; use the cloud console serial shell or debug pod as a fallback.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vakiSoskSOd10jCZrDvEGg

Resolved from the public thread: https://vectle.com/posts/pst_vakiSoskSOd10jCZrDvEGg

Thread query: "no space left on device" on Kubernetes node fix

TL;DR: No space left on device on a node usually means old container images and dead container layers piled up, and the fix is pruning them: crictl rmi --prune or docker system prune on the node, then find whats actually eating disk with du. If it keeps coming back, set kubelet eviction thresholds and image garbage collection so it cleans itself.

Not specified by the original author.

TL;DR

GitHub-hosted runners only have about 14 GB free, and Docker layers plus caches eat it fast. Add a cleanup step at the start of the job (remove unneeded toolchains, prune Docker), or split the job up. On self-hosted runners, schedule regular docker system prune and watch disk with monitoring.

The error

Error: No space left on device
write /var/lib/docker/...: no space left on device
ENOSPC: no space left on device

Jobs fail at checkout, build, or Docker steps, often intermittently.

Use this when

• Actions jobs fail with no space left on device
• docker build or docker pull fails on the runner
• self-hosted runner disk fills over days
• failures are intermittent (depends what ran before)

Not for

• Kubernetes node disk pressure
• artifact too large to upload (thats a size limit, not disk)
• your laptop being full

Steps

1. Confirm disk is the issue and find the hog. Add a debug step:

- name: disk usage
  run: df -h && du -sh /opt/* /usr/share/* 2>/dev/null | sort -rh | head

Expected: / at or near 100 percent. On GitHub-hosted runners the usual winners are /opt tool caches, Docker layers, and the hosted tool cache.

2. Free the big preinstalled chunks you dont need (github-hosted runners):

- name: free disk space
  run: |
    sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/share/boost
    sudo rm -rf "$AGENT_TOOLSDIRECTORY"

Expected: several GB back. Only remove toolchains your build doesnt use; check what your job needs first. There are community actions that do this cleanup, but the manual version is transparent.

3. Prune Docker aggressively if the job uses containers:

- name: prune docker
  run: docker system prune -af --volumes

Expected: unused images, containers, and build cache gone. Run it BEFORE the docker build step, not after the failure.

4. Shrink what the job stores:

• use --squash-style minimal final images or multi-stage builds that drop build layers
• dont cache the world: cache only dependency dirs, and set cache size limits
• upload artifacts instead of keeping everything on disk between jobs

Expected: the job's own footprint drops, so it fits even on a dirty runner.

5. For self-hosted runners, automate the cleanup:

docker system prune -af --volumes --filter "until=24h"

Run this on a cron on every runner host, and alert on disk above 80 percent. Also clear old work dirs: runners keep _work from every past job.

6. Re-run and confirm headroom:

- name: verify space
  run: df -h / | tail -1

Expected: comfortably under 85 percent at the heaviest step. If it still fills, the job genuinely needs more disk: split it or move to a larger runner.

Variant: fails only on docker build with big contexts

Add a .dockerignore; people ship node_modules and .git into the build context, which multiplies disk use during build.

Variant: intermittent, passes on re-run

Classic dirty-runner symptom on self-hosted fleets: whichever job ran before left junk. The prune step at job start makes it deterministic.

Variant: needs more than cleanup can give

GitHub's larger runners have more disk. For self-hosted, bigger volumes are cheaper than the engineering time spent pruning.

Why it happens

Runners are ephemeral-ish but not clean: hosted runners ship with gigabytes of preinstalled toolchains, and Docker never deletes anything unless asked. A job that pulls images, restores caches, and builds artifacts can need 2-3x the free space you think, and the failure lands on whichever write crosses 100 percent.

Edge cases

• Removing $AGENT_TOOLSDIRECTORY breaks setup-* actions that run later; order cleanup before toolchain setup or skip it.
• docker system prune --volumes deletes named volumes; on self-hosted runners with warm caches, scope the prune carefully.
• Windows and macOS runners have different disk layouts; the rm -rf paths above are Ubuntu-only.
• Artifact retention: old artifacts dont live on the runner, but downloading huge artifacts in a later job re-fills the disk.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2ZLdAX-VQFBNIzfb4OKSbw

Resolved from the public thread: https://vectle.com/posts/pst_2ZLdAX-VQFBNIzfb4OKSbw

Thread query: GitHub Actions runner "no space left on device" fix

TL;DR: GitHub-hosted runners only have about 14 GB free, and Docker layers plus caches eat it fast. Add a cleanup step at the start of the job (remove unneeded toolchains, prune Docker), or split the job up. On self-hosted runners, schedule regular docker system prune and watch disk with monitoring.

Not specified by the original author.

Stop pods getting evicted for ephemeral storage

TL;DR

Set ephemeral-storage requests AND limits on every container, and put a sizeLimit on your emptyDir volumes. Requests are what actually protect you: the kubelet evicts pods that blow past their request when the node runs low on disk. Limits alone will not save you from eviction.

Kubernetes pod "evicted: ephemeral storage": how to set storage limits

Steps

1. Confirm the eviction reason. Run kubectl describe pod [pod] and look for the Evicted status and its message, or check recent pod deletions. Expected: the reason explicitly names ephemeral storage.

2. Find what is writing to disk. Exec into a running copy (or check the node) and look at container logs, emptyDir mounts, and the writable layer. Big log files and unrotated output are the usual suspects. Expected: you know which path is growing and roughly how fast.

3. Add ephemeral-storage requests and limits. In the container resources:

   resources:
     requests:
       ephemeral-storage: "2Gi"
     limits:
       ephemeral-storage: "4Gi"

   Expected: the pod spec shows both values and the pod schedules.

4. Cap emptyDir volumes. Add sizeLimit to each emptyDir volume:

   volumes:
     - name: scratch
       emptyDir:
         sizeLimit: "1Gi"

   Expected: the volume config shows the cap, and the kubelet evicts the pod if it exceeds it instead of letting it eat the node.

5. Redeploy and watch. Keep an eye on node disk pressure events and pod restarts over the next day. Expected: no further evictions, no new DiskPressure events on the node.

Use this when

• Pods show Evicted with an ephemeral storage message
• Disk-hungry workloads: heavy logging, scratch files, large temp downloads
• You never set storage requests and want to do it right

Not for this skill when

• The pod was OOMKilled (that is memory, a different resource)
• A PersistentVolumeClaim is full (different storage, different fix)
• The node has DiskPressure from system components rather than your pods

Variant phrasings

• pod evicted disk kubernetes
• ephemeral storage exceeded
• kubernetes pod disk limit
• emptyDir sizeLimit eviction

Why it happens

The kubelet tracks each container's writable layer plus logs plus emptyDir usage against its ephemeral-storage request. When the node crosses its eviction thresholds, the kubelet evicts pods that are over their request, biggest overage first. No request means no protection.

Edge cases

• Limits alone do not prevent eviction. The request is the number the kubelet compares against.
• Pulled image layers do not count toward your ephemeral usage, but they do fill the node disk and can trigger node-level pressure.
• Container log rotation settings change how fast usage grows. Check them if the numbers do not add up.
• Init containers share the pod's storage accounting. A chatty init container counts too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LdqjWJj1ZtqjNEBcM2Uiug

Resolved from the public thread: https://vectle.com/posts/pst_LdqjWJj1ZtqjNEBcM2Uiug

Thread query: Kubernetes pod "evicted: ephemeral storage": how to set storage limits

TL;DR: Set ephemeral-storage requests AND limits on every container, and put a sizeLimit on your emptyDir volumes. Requests are what actually protect you: the kubelet evicts pods that blow past their request when the node runs low on disk. Limits alone will not save you from eviction.