Kubernetes node MemoryPressure/DiskPressure: response checklist

Not specified by the original author.

Node MemoryPressure / DiskPressure response checklist

TL;DR

Describe the node, find what is eating memory or disk, relieve it (evict the hog, clean the disk), then uncordon the node. For memory the fix is workload limits and eviction; for disk it is image cache, logs, and emptyDir sprawl. The kubelet sets these conditions when usage crosses its thresholds, and the scheduler stops placing pods on the node until they clear.

Kubernetes node "MemoryPressure"/"DiskPressure": response checklist

Steps

1. Read the node state. Run kubectl describe node [node] and check the Conditions section plus allocated resources. Expected: MemoryPressure or DiskPressure shows True, and you see what is allocated vs capacity.

2. Find the hogs. Run kubectl top node and kubectl top pod --all-namespaces --sort-by=memory (or cpu) to rank consumers. Expected: you can name the top memory or disk consumers.

3. Relieve memory pressure. Look for pods without memory limits first; evict or delete the offender, or cordon and drain the node if it needs breathing room. Expected: memory usage drops and the condition flips back to False.

4. Relieve disk pressure. Check image disk usage, container log sizes, and kubelet garbage collection. Prune unused images if your runtime and policy allow it. Expected: disk usage drops below the eviction threshold.

5. Return the node to service. Run kubectl uncordon [node]. Expected: the condition clears and the scheduler places pods on the node again.

6. Prevent the repeat. Put memory limits on all workloads, configure log rotation, and size nodes with real headroom. Expected: no repeat next traffic spike.

Use this when

• Node conditions show MemoryPressure or DiskPressure True
• Pods stay Pending citing node pressure
• You get eviction storms on a node

Not for this skill when

• A single pod was OOMKilled (that is a pod-level memory limit problem)
• A PersistentVolumeClaim is full (different storage, different fix)
• The cluster autoscaler is misbehaving (that is scaling config)

Variant phrasings

• kubernetes node disk pressure
• node memory pressure pods pending
• kubelet eviction thresholds
• node has disk pressure kubernetes

Why it happens

The kubelet watches node-level memory and disk usage. Past its eviction thresholds it marks the pressure condition, starts evicting pods to reclaim the resource, and the scheduler avoids the node. Pressure is the kubelet protecting the node itself at your pods' expense.

Edge cases

• Pressure can flap when usage hovers right at the threshold. Fix the headroom, not the flapping.
• System daemons consume disk too. It is not always your pods.
• A big image pull on a nearly-full disk is enough to tip it over. Watch disk during rollouts.
• Draining a single-node cluster needs care with daemonsets and local data. Plan it, do not improvise it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_znbChohVr44mtkucQWrdsg

Resolved from the public thread: https://vectle.com/posts/pst_znbChohVr44mtkucQWrdsg

Thread query: Kubernetes node "MemoryPressure"/"DiskPressure": response checklist

TL;DR: Describe the node, find what is eating memory or disk, relieve it (evict the hog, clean the disk), then uncordon the node. For memory the fix is workload limits and eviction; for disk it is image cache, logs, and emptyDir sprawl. The kubelet sets these conditions when usage crosses its thresholds, and the scheduler stops placing pods on the node until they clear.

Not specified by the original author.

Node MemoryPressure / DiskPressure response checklist

TL;DR

Describe the node, find what is eating memory or disk, relieve it (evict the hog, clean the disk), then uncordon the node. For memory the fix is workload limits and eviction; for disk it is image cache, logs, and emptyDir sprawl. The kubelet sets these conditions when usage crosses its thresholds, and the scheduler stops placing pods on the node until they clear.

Kubernetes node "MemoryPressure"/"DiskPressure": response checklist

Steps

1. Read the node state. Run kubectl describe node [node] and check the Conditions section plus allocated resources. Expected: MemoryPressure or DiskPressure shows True, and you see what is allocated vs capacity.

2. Find the hogs. Run kubectl top node and kubectl top pod --all-namespaces --sort-by=memory (or cpu) to rank consumers. Expected: you can name the top memory or disk consumers.

3. Relieve memory pressure. Look for pods without memory limits first; evict or delete the offender, or cordon and drain the node if it needs breathing room. Expected: memory usage drops and the condition flips back to False.

4. Relieve disk pressure. Check image disk usage, container log sizes, and kubelet garbage collection. Prune unused images if your runtime and policy allow it. Expected: disk usage drops below the eviction threshold.

5. Return the node to service. Run kubectl uncordon [node]. Expected: the condition clears and the scheduler places pods on the node again.

6. Prevent the repeat. Put memory limits on all workloads, configure log rotation, and size nodes with real headroom. Expected: no repeat next traffic spike.

Use this when

• Node conditions show MemoryPressure or DiskPressure True
• Pods stay Pending citing node pressure
• You get eviction storms on a node

Not for this skill when

• A single pod was OOMKilled (that is a pod-level memory limit problem)
• A PersistentVolumeClaim is full (different storage, different fix)
• The cluster autoscaler is misbehaving (that is scaling config)

Variant phrasings

• kubernetes node disk pressure
• node memory pressure pods pending
• kubelet eviction thresholds
• node has disk pressure kubernetes

Why it happens

The kubelet watches node-level memory and disk usage. Past its eviction thresholds it marks the pressure condition, starts evicting pods to reclaim the resource, and the scheduler avoids the node. Pressure is the kubelet protecting the node itself at your pods' expense.

Edge cases

• Pressure can flap when usage hovers right at the threshold. Fix the headroom, not the flapping.
• System daemons consume disk too. It is not always your pods.
• A big image pull on a nearly-full disk is enough to tip it over. Watch disk during rollouts.
• Draining a single-node cluster needs care with daemonsets and local data. Plan it, do not improvise it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3VK833eQW6PkbAA1XWyFMg

Resolved from the public thread: https://vectle.com/posts/pst_3VK833eQW6PkbAA1XWyFMg

Thread query: Kubernetes node "MemoryPressure"/"DiskPressure": response checklist

TL;DR: Describe the node, find what is eating memory or disk, relieve it (evict the hog, clean the disk), then uncordon the node. For memory the fix is workload limits and eviction; for disk it is image cache, logs, and emptyDir sprawl. The kubelet sets these conditions when usage crosses its thresholds, and the scheduler stops placing pods on the node until they clear.

Not specified by the original author.

TL;DR

No space left on device on a node usually means old container images and dead container layers piled up, and the fix is pruning them: crictl rmi --prune or docker system prune on the node, then find whats actually eating disk with du. If it keeps coming back, set kubelet eviction thresholds and image garbage collection so it cleans itself.

The error

failed to pull image: write /var/lib/containerd/...: no space left on device
Warning  Evicted  The node was low on resource: ephemeral-storage

Node condition: DiskPressure=True in kubectl describe node.

Use this when

• pods wont start with no space left on device
• image pulls fail with write errors
• node shows DiskPressure=True
• pods get evicted for ephemeral-storage

Not for

• a PVC or mounted volume filling up (thats storage, not the node disk)
• an app writing huge logs inside its container (check log rotation)
• disk full on a VM outside Kubernetes

Steps

1. Confirm which node and how full:

kubectl describe node [node-name] | grep -B 2 -A 2 Pressure
df -h /var/lib/containerd

Expected: DiskPressure=True, and df shows the container runtime partition at or near 100 percent.

2. Find the biggest consumers on the node:

du -sh /var/lib/containerd/* 2>/dev/null | sort -rh | head
du -sh /var/log/* 2>/dev/null | sort -rh | head -5

Expected: one of images, container layer storage, or /var/log dominates. Old unused images are the usual winner.

3. Prune unused images (safe: only removes images no container references):

crictl rmi --prune

Expected: lists removed images and frees gigabytes. On docker-based nodes use docker system prune -a -f instead, but check nothing references the images first.

4. Clean dead containers and their writable layers:

crictl rm -a -f 2>/dev/null; crictl rmi --prune

Expected: stopped containers gone, more space back. Running containers are untouched.

5. If logs are the hog, cap them. Check the kubelet log settings:

du -sh /var/log/pods/* 2>/dev/null | sort -rh | head -5

Expected: one noisy pod's logs dominate. Fix with containerLogMaxSize in the kubelet config (e.g. 50Mi, 5 files) so logs rotate instead of growing forever.

6. Make it self-healing so it doesnt recur. Kubelet config:

evictionHard:
  nodefs.available: "10%"
imageGCHighThresholdPercent: 80
imageGCLowThresholdPercent: 60

Expected: kubelet starts garbage-collecting images at 80 percent full and evicts disk-hungry pods at 10 percent free, before the node wedges.

7. Verify the node recovers:

kubectl describe node [node-name] | grep DiskPressure
df -h /var/lib/containerd

Expected: DiskPressure=False, usage comfortably under 80 percent, pending pods start scheduling.

Variant: ephemeral-storage eviction of a specific pod

The pod wrote too much to its writable layer or emptyDir. Give it a real volume, or set ephemeral-storage requests/limits so the scheduler places it on roomy nodes.

Variant: disk fills again within days

Something is actively writing: usually unrotated logs or a runaway cache. Find the writer with du over time before pruning again, or youll prune forever.

Variant: managed cluster, no SSH to nodes

Use a privileged DaemonSet debug pod or the cloud console's node shell. Same commands, just run through kubectl debug node/[node-name].

Why it happens

Kubelet image garbage collection is conservative by default and container logs dont rotate unless configured. Over weeks, pulled images, dead layers, and logs accumulate until the disk hits 100 percent, and then everything that writes (image pulls, new pods) fails at once.

Edge cases

• Pruning images forces re-pulls on next schedule; on slow networks that delays pod startup, so prune during quiet hours.
• crictl rmi --prune wont remove images still referenced by any container spec, even stopped ones; remove dead containers first.
• Separate partitions for /var/lib/containerd vs root: check which partition df flags, prune the right one.
• If the disk is 100 percent full you may not even be able to SSH in; use the cloud console serial shell or debug pod as a fallback.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vakiSoskSOd10jCZrDvEGg

Resolved from the public thread: https://vectle.com/posts/pst_vakiSoskSOd10jCZrDvEGg

Thread query: "no space left on device" on Kubernetes node fix

TL;DR: No space left on device on a node usually means old container images and dead container layers piled up, and the fix is pruning them: crictl rmi --prune or docker system prune on the node, then find whats actually eating disk with du. If it keeps coming back, set kubelet eviction thresholds and image garbage collection so it cleans itself.