how to clean up evicted pods filling node disk

TL;DR

Evicted pods are dead but their API objects linger, and hundreds of them slow kubectl and clutter scheduling decisions. List them, delete them in bulk (they are already terminated, so deletion is safe), then fix the recurrence by setting pod eviction thresholds or adding capacity. Deleting evicted pod objects never affects running workloads.

Error / query

how to clean up evicted pods filling node disk

Use this skill when

• kubectl get pods shows hundreds of Evicted pods
• Node disk pressure correlates with dead pod accumulation
• kubectl list calls are slow and namespaces are full of terminated pods
• Evictions keep recurring after each cleanup

Not for this skill when

• Running pods are filling disk with logs or data (clean the workload)
• Pods are being OOMKilled (sizing problem, not eviction cleanup)
• etcd itself is full (compaction/defrag, different fix)

Steps

Step 1: Count evicted pods per namespace

kubectl get pods -A --field-selector=status.phase=Failed -o json | jq -r '.items[] | select(.status.reason=="Evicted") | .metadata.namespace' | sort | uniq -c | sort -rn

Expected: a count per namespace. This tells you the scale and whether one namespace's workload is the eviction magnet.

Step 2: Delete evicted pods in bulk

kubectl get pods -A --field-selector=status.phase=Failed -o json | jq -r '.items[] | select(.status.reason=="Evicted") | "\(.metadata.namespace) \(.metadata.name)"' | while read ns name; do kubectl delete pod "$name" -n "$ns"; done

Expected: evicted pod objects disappear. These pods are already terminated, so deletion only removes the API object; nothing running is touched.

Step 3: Find why evictions keep happening

kubectl describe node [node-name] | grep -A10 "Pressure\|Allocatable"

Expected: the pressure condition (usually DiskPressure or MemoryPressure) and how tight allocatable resources are. Cleanup treats the symptom; pressure is the disease.

Step 4: Prevent recurrence

kubectl get pods -n [namespace] [workload] -o jsonpath='{.spec.template.spec.containers[0].resources}{"\n"}'

Expected: you see whether the evicted workloads have realistic requests. Set kubelet eviction thresholds (--eviction-hard) so the kubelet acts earlier and more gracefully, give chronic offenders real requests, or add node capacity.

Variant phrasings

"delete evicted pods kubernetes"

Step 2 is the bulk delete. It is safe: evicted pods are already dead.

"kubernetes evicted pods piling up"

Cleanup (step 2) plus prevention (step 4). Piling up means the pressure that caused them is still there.

Why it happens

When the kubelet evicts pods under resource pressure, the pod objects stay in the API with status Failed and reason Evicted until something deletes them. Controllers do not clean them up. At scale they accumulate into the thousands, slowing list operations and masking fresh failures in a sea of dead pods.

Edge cases and pitfalls

• Do not delete pods by phase Failed blindly; some Failed pods are completed Jobs you want to inspect. Filter on reason==Evicted.
• If evictions come from DiskPressure caused by image layers or logs, deleting pod objects will not free disk; clean the underlying cause.
• CronJobs create many short-lived pods; their evicted remains pile fastest, so scope the cleanup per namespace.
• A bulk delete during an active pressure event just clears the view; new evictions will replace them until capacity or requests are fixed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DjoSITH6zWLvM1tvR3GcDQ