OOMKilled kubernetes

Not specified by the original author.

TL;DR

An OOMKilled pod means a container exceeded its memory limit (or the node ran dry). Check kubectl describe pod for the last state, find the culprit with memory usage stats, then fix by raising limits or fixing the leak. The kill almost always comes from a limit you set, not the node itself.

The error

Last State: Terminated
Reason: OOMKilled
Exit Code: 137

Also shows up as: kubectl get pods shows OOMKilled under STATUS, or events say Out of memory: Kill process from the kernel.

Use this when

• pod status is OOMKilled or exit code 137
• a pod restarts repeatedly under load
• you suspect a memory leak in a container
• requests/limits were set long ago and never revisited

Not for

• pods killed by liveness probes (different reason string)
• node-level memory pressure where the kubelet evicts pods (check node events instead)
• CPU problems; memory and CPU limits behave very differently

Steps

1. Confirm the OOMKilled and which container died:

kubectl describe pod [pod-name] -n [namespace]

Expected: under Containers, Last State: Terminated, Reason: OOMKilled, Exit Code: 137. Note the container name and the memory request/limit in the same block.

2. Check how the pod is behaving over time:

kubectl get pod [pod-name] -n [namespace] -w
kubectl top pod [pod-name] -n [namespace]

Expected: kubectl top shows current memory vs the limit. If usage climbs steadily toward the limit before each restart, its a leak or growth, not a one-off spike.

3. See if the node itself is under pressure (rules out node-level kills):

kubectl describe node [node-name] | grep -A 5 Conditions
kubectl get events -n [namespace] --sort-by=.metadata.creationTimestamp | grep -i memory

Expected: node Conditions show MemoryPressure=False if the kill came from the container limit. Events will say which limit was exceeded.

4. Check for a leak pattern inside the app:

kubectl logs [pod-name] -n [namespace] --previous | tail -100

Expected: look for heap dumps, GC errors, or the app logging its own memory stats right before dying. Repeat OOM kills right after startup point at a startup burst; kills after hours point at a slow leak.

5. Fix it. Pick one:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

Raise the limit to something sane (a headroom of ~20 percent over observed peak), set a request so the scheduler places the pod on a node with room, and deploy. If the app leaks, fix the leak in code instead of endlessly raising the limit.

6. Apply and verify the pod stays up:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/[name] -n [namespace]

Expected: rollout completes, new pods show STATUS Running, no OOMKilled in describe output after a full load cycle.

Variant: OOMKilled but the container limit looks fine

The node ran out of memory and the kernel picked your pod. Check node MemoryPressure, lower total commitments on the node, or add requests so the scheduler stops overpacking.

Variant: exit code 137 without OOMKilled in the reason

Same fix path. 137 = SIGKILL, which in Kubernetes is almost always the OOM killer. Confirm via kubectl describe pod.

Variant: Java app OOMKilled with plenty of headroom in the limit

JVM heap defaults plus off-heap can exceed the container limit. Set -Xmx to about 75 percent of the container limit so heap plus overhead fits.

Why it happens

Kubernetes enforces memory limits with cgroups. When a container touches more memory than its limit, the kernel OOM killer terminates it instantly, no graceful shutdown. Exit code 137 is the signature. Most of the time the limit was set too low for real workload, or the app has a slow leak nobody noticed because restarts masked it.

Edge cases

• No limit set at all means the container can eat the node and the kernel picks victims by oom_score; always set limits.
• Bursty workloads (batch jobs, image processing) need limits far above the request; use a VerticalPodAutoscaler in recommendation mode to find real numbers.
• OOMKilled during init or startup often means the request is fine but the limit is below the startup peak; watch kubectl top during a fresh deploy.
• If many pods die at once, suspect node pressure or a bad deploy, not individual leaks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tbJdavdXVNdMEZhjCgDLMA

Resolved from the public thread: https://vectle.com/posts/pst_tbJdavdXVNdMEZhjCgDLMA

Thread query: Kubernetes OOMKilled: how to find the memory hog

TL;DR: An OOMKilled pod means a container exceeded its memory limit (or the node ran dry). Check kubectl describe pod for the last state, find the culprit with memory usage stats, then fix by raising limits or fixing the leak. The kill almost always comes from a limit you set, not the node itself.

Not specified by the original author.

TL;DR

Raise memory limits with a rolling update, keep requests honest so the scheduler can place the pod, and leave headroom between request and limit for bursts. Do it in small steps and watch actual usage with kubectl top or a VerticalPodAutoscaler in recommendation mode. Safe means no scheduling surprises and no evictions.

The query

how to increase Kubernetes pod memory limits safely

You want bigger limits without breaking scheduling, causing evictions, or masking a leak.

Use this when

• pods are OOMKilled and need bigger limits
• usage graphs show the pod riding near its limit
• youre right-sizing requests and limits for a workload
• a VPA recommendation suggests new values

Not for

• CPU limits (CPU throttles, memory kills; different tuning)
• adding RAM to the nodes themselves
• debugging why a pod got OOMKilled (fix the cause first)

Steps

1. Measure real usage before touching anything:

kubectl top pod [pod-name] -n [namespace] --containers

Expected: current usage per container. For history, check your metrics backend (Prometheus container_memory_working_set_bytes). Size from peaks, not averages.

2. Set request to the steady-state usage and limit to peak plus headroom:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

Expected rule of thumb: request near the p50, limit near the p99 plus 20 percent. The request is what the scheduler reserves; the limit is where the OOM killer strikes.

3. Apply with a rolling update, never by editing a live pod (pods are immutable):

kubectl apply -f deployment.yaml
kubectl rollout status deployment/[name] -n [namespace]

Expected: new pods come up with the new limits, old pods drain. kubectl describe pod on a new pod shows the updated values.

4. Confirm the pod actually fits on the nodes:

kubectl describe node | grep -A 5 "Allocated resources"

Expected: the node's allocated memory still has room for the new requests. If raising requests makes pods unschedulable, the cluster needs more capacity, not smaller requests.

5. Watch one full load cycle before calling it done:

kubectl top pod -n [namespace] -l app=[app] --containers

Expected: peak usage stays comfortably under the new limit. If it touches the limit again, either raise further or fix the growth in the app.

6. For ongoing sizing, run a VPA in recommendation mode:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: [app]-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: [app]
  updatePolicy:
    updateMode: "Off"

Expected: kubectl describe vpa [app]-vpa shows recommended requests. Apply the recommendations manually on your own schedule.

Variant: bursty workload (batch jobs, image processing)

Keep requests low so scheduling stays easy, set limits high for the bursts. Wide request/limit gaps are fine for bursty work and terrible for steady-state services.

Variant: Java/JVM apps

Set -Xmx to about 75 percent of the new limit so heap plus off-heap fits. Raising the container limit without raising Xmx just moves the waste.

Variant: limit raised but pod still OOMKilled

The rollout may not have replaced the pod (check the pod age and the limit in describe), or the app grows unboundedly. Verify the new limit is live before raising again.

Why it happens

Memory limits are enforced by cgroups: cross the line and the kernel kills the container instantly. Requests control scheduling: the scheduler wont place a pod where the request doesnt fit. Raising limits without watching requests either leaves pods unschedulable (request too big) or lets one pod starve its neighbors (request too small, limit huge).

Edge cases

• LimitRange in the namespace can cap or default your values; check kubectl describe limitrange -n [namespace] if your numbers dont stick.
• VPA in Auto mode restarts pods to apply recommendations; use Off or Initial during business hours.
• Memory QoS: pods with requests equal to limits get Guaranteed class and are evicted last; burstable pods are evicted first under node pressure.
• Dont raise limits to mask a leak; if usage climbs forever, the code needs fixing, not the YAML.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tvOQlWmx38kr1V5oDJ5PpQ

Resolved from the public thread: https://vectle.com/posts/pst_tvOQlWmx38kr1V5oDJ5PpQ

Thread query: how to increase Kubernetes pod memory limits safely

TL;DR: Raise memory limits with a rolling update, keep requests honest so the scheduler can place the pod, and leave headroom between request and limit for bursts. Do it in small steps and watch actual usage with kubectl top or a VerticalPodAutoscaler in recommendation mode. Safe means no scheduling surprises and no evictions.

Not specified by the original author.

TL;DR

Set requests from measured steady-state usage plus headroom, and set limits from measured peak plus a smaller buffer: requests around p50 usage times 1.2, limits around p99 usage times 1.3. Start new workloads with a generous limit and tighten after a week of data. Requests that are too low cause evictions; limits that are too low cause OOMKills; get both from data, not guesses.

Error / query

pod memory requests vs limits: a sizing recipe

Use this skill when

• You are writing resource blocks for a new deployment
• Pods get OOMKilled or evicted and the numbers look arbitrary
• You want a standard requests/limits policy across services
• VPA or Goldilocks-style recommendations need a sanity check

Not for this skill when

• A pod is actively crash-looping on OOMKilled (fix the immediate limit first)
• You are sizing CPU (different dynamics: throttling, not killing)
• The node itself is undersized (capacity planning, not pod sizing)

Steps

Step 1: Measure actual memory usage over a representative window

kubectl top pod -n [namespace] -l app=[app] --containers

Expected: current per-container usage. For the recipe you need history, so pull the same metric from Prometheus (container_memory_working_set_bytes) over 7+ days covering peak traffic, deploys, and batch jobs.

Step 2: Set the request from steady state plus headroom

resources:
  requests:
    memory: "[p50-usage x 1.2]"

Expected: requests land near typical usage with 20 percent headroom. The scheduler uses requests for placement, so honest requests keep nodes from being overcommitted into eviction territory.

Step 3: Set the limit from peak plus a smaller buffer

resources:
  limits:
    memory: "[p99-usage x 1.3]"

Expected: limits sit above real peaks with 30 percent buffer. The limit is a kill switch, not a target: it should almost never be hit, but must absorb genuine spikes (deploys, traffic bursts, GC pauses).

Step 4: Deploy, watch for a week, then tighten

kubectl get events -n [namespace] --field-selector reason=OOMKilling --sort-by=.lastTimestamp

Expected: zero OOMKilling events and no evictions. If clean, you can tighten the limit toward p99 x 1.2. If you see kills, the peak measurement missed something; widen the measurement window before raising blindly.

Variant phrasings

"kubernetes memory request limit best practice"

Requests from typical usage, limits from peak usage, both from measured data. Steps 2 and 3 are the recipe.

"how to set pod memory limits"

Measure peak first (step 1), then limit at peak x 1.3 (step 3). Never copy limits from another service.

Why it happens

Requests and limits do different jobs: requests drive scheduling and eviction priority, limits drive the OOM killer. Teams that set both to the same guess either waste capacity (limits too high to matter) or get killed on normal spikes (limits too tight). Measuring separates the two jobs and gives each number a reason.

Edge cases and pitfalls

• JVM and other runtimes need heap flags set below the limit, or the runtime will happily allocate past it and get OOMKilled.
• Init containers with big memory needs can block scheduling; their requests count during scheduling even though they are short-lived.
• Setting request equal to limit gives Guaranteed QoS (evicted last); leaving request unset gives BestEffort (evicted first). Choose deliberately.
• Memory usage that grows unboundedly is a leak, not a sizing problem; no limit recipe fixes a leak, it just delays the kill.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst2HStBYVYJFU95Lx0ew6pw

Resolved from the public thread: https://vectle.com/posts/pst2HStBYVYJFU95Lx0ew6pw

Thread query: pod memory requests vs limits: a sizing recipe

TL;DR: Set requests from measured steady-state usage plus headroom, and set limits from measured peak plus a smaller buffer: requests around p50 usage times 1.2, limits around p99 usage times 1.3. Start new workloads with a generous limit and tighten after a week of data. Requests that are too low cause evictions; limits that are too low cause OOMKills; get both from data, not guesses.