Kubernetes pod "evicted: ephemeral storage": how to set storage limits

Stop pods getting evicted for ephemeral storage

TL;DR

Set ephemeral-storage requests AND limits on every container, and put a sizeLimit on your emptyDir volumes. Requests are what actually protect you: the kubelet evicts pods that blow past their request when the node runs low on disk. Limits alone will not save you from eviction.

Kubernetes pod "evicted: ephemeral storage": how to set storage limits

Steps

1. Confirm the eviction reason. Run kubectl describe pod [pod] and look for the Evicted status and its message, or check recent pod deletions.

Expected: the reason explicitly names ephemeral storage.

1. Find what is writing to disk. Exec into a running copy (or check the node) and look at container logs, emptyDir mounts, and the writable layer. Big log files and unrotated output are the usual suspects.

Expected: you know which path is growing and roughly how fast.

1. Add ephemeral-storage requests and limits. In the container resources:

   resources:
     requests:
       ephemeral-storage: "2Gi"
     limits:
       ephemeral-storage: "4Gi"

Expected: the pod spec shows both values and the pod schedules.

1. Cap emptyDir volumes. Add sizeLimit to each emptyDir volume:

   volumes:
     - name: scratch
       emptyDir:
         sizeLimit: "1Gi"

Expected: the volume config shows the cap, and the kubelet evicts the pod if it exceeds it instead of letting it eat the node.

1. Redeploy and watch. Keep an eye on node disk pressure events and pod restarts over the next day.

Expected: no further evictions, no new DiskPressure events on the node.

Use this when

• Pods show Evicted with an ephemeral storage message
• Disk-hungry workloads: heavy logging, scratch files, large temp downloads
• You never set storage requests and want to do it right

Not for this skill when

• The pod was OOMKilled (that is memory, a different resource)
• A PersistentVolumeClaim is full (different storage, different fix)
• The node has DiskPressure from system components rather than your pods

Variant phrasings

• pod evicted disk kubernetes
• ephemeral storage exceeded
• kubernetes pod disk limit
• emptyDir sizeLimit eviction

Why it happens

The kubelet tracks each container's writable layer plus logs plus emptyDir usage against its ephemeral-storage request. When the node crosses its eviction thresholds, the kubelet evicts pods that are over their request, biggest overage first. No request means no protection.

Edge cases

• Limits alone do not prevent eviction. The request is the number the kubelet compares against.
• Pulled image layers do not count toward your ephemeral usage, but they do fill the node disk and can trigger node-level pressure.
• Container log rotation settings change how fast usage grows. Check them if the numbers do not add up.
• Init containers share the pod's storage accounting. A chatty init container counts too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LdqjWJj1ZtqjNEBcM2Uiug