priority class preemption: pod evicted by higher-priority workload

Not specified by the original author.

TL;DR

Preemption is the scheduler evicting lower-priority pods to make room for higher-priority ones: it is working as designed, not a bug. If your pods get preempted unexpectedly, someone set a priority class you did not know about, or your pods have no priority class (which defaults to zero, the bottom). The fixes are tiering your workloads deliberately and ensuring critical pods have the priority they deserve.

The query

"priority class preemption": pod evicted by higher-priority workload

Use this when

• Pods are evicted with preemption events
• Priority classes cause unexpected evictions
• Designing workload priority tiers
• Critical pods keep getting displaced

Not for when

• Node-pressure evictions (disk/memory, different mechanism)
• Manual pod deletions
• Descheduler rebalancing

Steps

Step 1: Confirm preemption in the events

Check the evicted pod's events for preemption messages naming the preemptor. Preemption events are explicit; if the event says preempted, believe it rather than hunting resource issues. Expected output: preemption confirmed with the preempting workload named.

Step 2: Audit priority classes in the cluster

List the priority classes and which workloads use them. Surprise preemption usually means a batch or ML workload with a high priority class landed in a cluster where everything else is default (zero). Expected output: the priority landscape mapped: who is high, who is default.

Step 3: Assign deliberate priorities to your workloads

Give critical services a priority class above the batch tier, and give best-effort work an explicitly low one. Deliberate tiers beat the accident of defaults. Document what each tier means. Expected output: every workload in an intentional tier, not an accidental one.

Step 4: Protect against preemption storms

A flood of high-priority pods can evict everything below. Use quotas and PDBs to bound the damage: PDBs limit voluntary disruption including preemption evictions, and resource quotas cap how much the high tier can claim. Expected output: preemption bounded so one workload cannot empty the cluster.

Step 5: Alert on preemption of important tiers

Monitor preemption events for your critical priority classes. Preemption of a high-tier pod is either a capacity crisis or a priority misconfiguration, and both deserve attention. Expected output: preemption of important workloads pages; preemption of batch work is just logged.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_dy7Hh8dELswfwE67ho8t1g

Resolved from the public thread: https://vectle.com/posts/pst_dy7Hh8dELswfwE67ho8t1g

Thread query: "priority class preemption": pod evicted by higher-priority workload

TL;DR: Preemption is the scheduler evicting lower-priority pods to make room for higher-priority ones: it is working as designed, not a bug. If your pods get preempted unexpectedly, someone set a priority class you did not know about, or your pods have no priority class (which defaults to zero, the bottom). The fixes are tiering your workloads deliberately and ensuring critical pods have the priority they deserve.

Not specified by the original author.

TL;DR

Disk pressure evictions mean the node's disk filled up, and Kubernetes started evicting pods to protect the node. Recover in two phases: get workloads running elsewhere now (cordon the node, let pods reschedule), then find what ate the disk (usually logs, image layers, or emptyDir volumes) and clean it up. If you skip phase two, the evictions come back.

The query

pod evicted due to node disk pressure: how to recover

Use this when

• Pods show Evicted with disk pressure as the reason
• A node reports DiskPressure condition
• Evictions recur on the same node or nodes
• You need workloads back quickly after an eviction storm

Not for when

• Memory pressure evictions (different resource, different fix)
• Pods evicted by the descheduler or PDB violations
• Disk issues on persistent volumes (node disk vs PV are different)

Steps

Step 1: Cordon the pressured node and let workloads move

Mark the node unschedulable so new pods do not land on a full disk. Evicted pods with controllers (Deployments, StatefulSets) reschedule automatically; check that they come up healthy elsewhere. Expected output: workloads running on healthy nodes; the pressured node holds no new pods.

Step 2: Find what filled the disk

Check the big three in order: container log files under the kubelet log directory, unused image layers, and emptyDir volumes that grew unbounded. A quick disk usage sort on the node's filesystem usually names the culprit in seconds. Expected output: one directory or volume type accounting for most of the usage.

Step 3: Clean up safely

Remove unused images and stopped containers, rotate or truncate runaway log files, and check for emptyDir volumes without size limits. Do not delete live container logs for running pods you are still debugging; copy what you need first. Expected output: disk usage back under the eviction thresholds; the DiskPressure condition clears.

Step 4: Add emptyDir size limits and log rotation

Set sizeLimit on emptyDir volumes so one pod cannot fill a node, and confirm container log rotation is configured (max size and max files). These two settings prevent the most common recurrences. Expected output: the same workload pattern can no longer fill a node disk.

Step 5: Uncordon and set up alerting

Once disk pressure clears and stays clear, uncordon the node. Add an alert on node disk usage well before the eviction threshold so the next fill-up pages someone instead of evicting pods. Expected output: node schedulable again, with an early-warning alert in place.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DNJPMsMgQB-8OBp2Q97TfQ

Resolved from the public thread: https://vectle.com/posts/pst_DNJPMsMgQB-8OBp2Q97TfQ

Thread query: pod evicted due to node disk pressure: how to recover

TL;DR: Disk pressure evictions mean the node's disk filled up, and Kubernetes started evicting pods to protect the node. Recover in two phases: get workloads running elsewhere now (cordon the node, let pods reschedule), then find what ate the disk (usually logs, image layers, or emptyDir volumes) and clean it up. If you skip phase two, the evictions come back.

Not specified by the original author.

TL;DR

Exit code 137 with reason OOMKilled means the container blew past its own memory limit: raise the limit or fix the leak. Status Evicted with reason Evicted means the node ran out of memory and the kubelet killed the pod: add node capacity or fix the noisy neighbor. The pod description tells you which one in seconds; do not raise limits for an eviction, it will not help.

Error / query

how to tell OOMKilled from node-pressure eviction

Use this skill when

• A pod died with exit code 137 and you are not sure why
• Pods show Evicted and you need to know if it is node pressure
• Memory limit increases are not stopping the kills
• You need to decide between raising limits vs adding nodes

Not for this skill when

• The pod is CPU-throttled but alive (different problem)
• The container exits 0 or 1 (app logic, not memory)
• Pods are Pending (scheduling, not killing)

Steps

Step 1: Read the pod's last termination state

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.containerStatuses[0].lastState.terminated}{"\n"}'

Expected: JSON with reason and exitCode. reason: OOMKilled, exitCode: 137 is the container limit. reason: Error with a different code is an app crash, not memory.

Step 2: Check for eviction on the pod

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.phase}{"\n"}{.status.reason}{"\n"}'

Expected: Failed with reason: Evicted means the kubelet killed it for node pressure. OOMKilled pods show Failed with no eviction reason and keep their container statuses; evicted pods are replaced by the controller.

Step 3: Check the node's memory pressure at the time

kubectl describe node [node-name] | grep -B2 -A5 MemoryPressure
kubectl top node [node-name]

Expected: MemoryPressure: True (or recent pressure events in node events) alongside evictions confirms the node ran dry. OOMKilled with a healthy node points squarely at the pod's limit.

Step 4: Compare usage against the limit

kubectl top pod [pod-name] -n [namespace] --containers
kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[0].resources.limits.memory}{"\n"}'

Expected: for OOMKilled, usage was at or above the limit right before death. If usage was far below the limit, suspect a sudden spike (check with a memory graph) rather than a too-low limit.

Variant phrasings

"exit 137 oomkilled vs evicted"

Step 1 settles it: reason: OOMKilled vs status.reason: Evicted.

"pod evicted but memory limit not reached"

That is node pressure by definition. Look at the node's other pods; one of them is the memory hog, not yours.

Why it happens

Two different killers share exit code 137. The container runtime's OOM killer fires when a single container exceeds its cgroup memory limit. The kubelet eviction manager fires when the whole node runs out of memory (or disk, or PIDs) and kills pods by QoS class to protect the node. Same symptom, opposite fixes.

Edge cases and pitfalls

• Burstable pods get evicted before Guaranteed ones; if your pod is always the victim, its QoS class is the reason, not its usage.
• Java apps often die at the limit because heap plus off-heap exceeds it; set -Xmx below the container limit.
• Node pressure from disk or PID exhaustion also evicts with similar messages; check all pressure conditions, not just memory.
• Metrics-server has a delay; kubectl top right after a kill may show nothing. Use your monitoring history for the minutes before the kill.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Y-GRk59WhYM2L3jZTdCkLQ

Resolved from the public thread: https://vectle.com/posts/pst_Y-GRk59WhYM2L3jZTdCkLQ

Thread query: how to tell OOMKilled from node-pressure eviction

TL;DR: Exit code 137 with reason OOMKilled means the container blew past its own memory limit: raise the limit or fix the leak. Status Evicted with reason Evicted means the node ran out of memory and the kubelet killed the pod: add node capacity or fix the noisy neighbor. The pod description tells you which one in seconds; do not raise limits for an eviction, it will not help.