Nextflow run fails with Process terminated with an error exit status (137)

Not specified by the original author.

TL;DR

Exit code 137 is 128 plus SIGKILL. Check the container's last termination reason: OOMKilled means your container crossed its memory limit (fix the limit or the leak), while a pod-level Evicted status means the node ran out of memory (fix node capacity or pod priority).

Verbatim error

State:          Terminated
  Reason:       OOMKilled
  Exit Code:    137

Steps

1. Read the termination state of the dead container.

kubectl get pod [POD] -n [NAMESPACE] -o yaml

Expected: in lastState.terminated you see either Reason OOMKilled with Exit Code 137 (container killed for exceeding its limit) or the pod itself in phase Failed with reason Evicted (node-pressure eviction). This one field decides the whole fix.

2. If the reason is OOMKilled, compare the limit against actual usage.

kubectl top pod [POD] -n [NAMESPACE]
kubectl describe pod [POD] -n [NAMESPACE]

Expected: describe shows the memory limit; top shows usage near or above it just before death. If there is no limit set, the container used enough to trigger the node's OOM killer anyway.

3. Fix an OOMKilled container: raise the memory limit if the usage is legitimate, or fix the leak if usage grows without bound.

kubectl -n [NAMESPACE] edit deployment [WORKLOAD]

Expected: after raising limits under resources, the new pods survive the same workload. For a leak, take a heap dump or profile before raising limits blindly; bigger limits just delay the next kill.

4. If the pod was Evicted instead, check node memory pressure at the time.

kubectl describe node [NODE]
kubectl get events -A --field-selector reason=Evicted

Expected: the node shows MemoryPressure in its conditions around the eviction time, and the event names the pod. Eviction picks pods by priority and by how far usage exceeds requests.

5. Fix eviction risk: set realistic memory requests so the scheduler places pods honestly, add a priorityClassName to workloads that must survive, and make sure the node pool has headroom.

kubectl get priorityclass

Expected: your critical workloads carry a PriorityClass above best-effort. Without priority, any burstable pod is eviction fodder when the node fills up.

When to use

• A container terminates with exit code 137
• You need to know whether to raise a limit or add node capacity
• The 137s started after a deploy changed memory limits or traffic grew

When not to use

• Exit code 143 (that is SIGTERM, usually a graceful shutdown or liveness probe kill)
• The pod is in CrashLoopBackOff with exit code 1; that is an app error, not memory
• kubectl logs shows nothing because the pod was evicted; check events, not logs

Why it happens

The kernel OOM killer sends SIGKILL when memory accounting says something must die: either the container's cgroup exceeded its limit (OOMKilled), or the whole node ran out and the kubelet evicted pods to relieve pressure (Evicted). Both surface as 137 because SIGKILL cannot be caught; only the termination reason tells them apart.

Edge cases

• No memory limit plus a memory leak: the container grows until the node OOM killer picks it; it looks like a node problem but the fix is in the app.
• Init containers share the pod's fate but have their own limits; check their lastState too.
• JVM and similar runtimes need heap settings below the container limit; the runtime happily allocates past the heap into native memory and gets OOMKilled.
• Swap enabled on the node delays eviction but slows pods first; check node swap settings if eviction timing looks off.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CrGfnxyIFjpaLBNPbnqicw

Resolved from the public thread: https://vectle.com/posts/pst_CrGfnxyIFjpaLBNPbnqicw

Thread query: kubectl logs exit code 137: OOMKilled vs node-pressure eviction

TL;DR: Exit code 137 is 128 plus SIGKILL. Check the container's last termination reason: OOMKilled means your container crossed its memory limit (fix the limit or the leak), while a pod-level Evicted status means the node ran out of memory (fix node capacity or pod priority).

Not specified by the original author.

TL;DR

Stop rerunning it locally; your machine has the RAM and CI does not. Check the CI logs for the OOM killer (exit code 137 is the tell), then either give the job a bigger runner or slim the test's memory footprint. The reruns were never going to reproduce a memory ceiling they do not have.

The exact query

my agent kept re-running a failing test locally where it passed  -  the CI runner had 2GB RAM and the test needed 4

Steps

1. Stop the local reruns and read the CI logs for memory evidence: look for exit code 137 (OOM kill), "Killed" messages, GC thrashing warnings, or the job dying at the same memory-heavy step every time. One 137 in the log ends the debate. Expected: A concrete memory signal from CI, for example "container exited 137 during the fixture that loads the 3GB dataset".

2. Measure the test's actual footprint: run it locally under memory observation (container with a 2GB limit, or /usr/bin/time -v, or the language's memory profiler) and record the peak. Compare against the CI runner's limit. Expected: Numbers, for example "peak 3.8GB vs CI limit 2GB". The gap is the diagnosis.

3. Choose the fix deliberately: if the test's memory use is legitimate (large dataset, heavy browser), raise the CI job's memory to fit it and document the requirement next to the test. If the footprint is accidental (loading the whole dataset for a test that needs 10 rows, a leak across tests), slim the test instead. Expected: Either the runner is sized to the test with the requirement documented, or the test is slimmed to fit the runner. No silent mismatch remains.

4. Apply the fix and verify under the real limit: run the test in a container capped at the CI memory limit. It must pass with headroom, not exactly at the edge. Expected: Green under the cap with margin. A test that passes at 1.99GB on a 2GB runner will fail the next time anything shifts.

5. Add a memory guardrail: a CI step that reports peak memory per test (or per job), and an alert when a test's footprint crosses a threshold of the runner's limit. Future memory pressure shows up as a trend, not a mystery failure. Expected: The next test that outgrows its runner is caught by the trend line before it starts failing.

Use this when

• A test fails only in CI and always passes locally
• CI logs show exit 137, "Killed", or the job dying at a memory-heavy step
• The CI runner has much less RAM than the machines used for reproduction
• An agent keeps rerunning locally instead of checking CI's memory ceiling

Not for this skill when

• There is no memory signal in the failure (no 137, no kill, plenty of headroom measured)
• The test is heavy by accepted design and the runner was already budgeted for it (then something else regressed; check what changed)
• The failure is a hard error (assertion, exception) rather than a death or slowdown

Variant phrasings

CI container exited with code 137 during tests

That is the OOM killer. Find the memory hog, size the runner or slim the test.

test passes locally but gets killed in CI

Memory ceiling difference. Measure the footprint, compare to the CI limit.

how to tell if CI test failure is OOM

Exit code 137, "Killed" in the log, death at the same heavy step every time. Then measure peak vs the runner limit.

Why it happens

Local machines are generous (16 to 64GB) and CI runners are not (2 to 4GB is common on free tiers). A test that casually uses 4GB never notices the difference locally, but in CI the OOM killer notices. The agent reproduces where the constraint does not exist, so every rerun passes and the investigation stalls. Memory pressure also degrades gracefully before it kills: GC thrashing makes tests slow and flaky before the 137 arrives, which sends the agent chasing timing ghosts.

Edge cases

• Memory leaks across tests can make the killer strike a test that is not itself heavy; the footprint measurement must cover the whole worker process, not just the single test in isolation.
• Parallel runners multiply memory: 4 workers each using 1GB need 4GB plus overhead. Size for workers times per-test peak, not a single test.
• Some languages over-commit or the container limit differs from the CI-advertised limit. Measure inside the actual CI container image, not just on your laptop with a limit flag.
• Swap can mask the problem locally (slow but alive) while CI has no swap (dead). Disable swap in your local reproduction for fidelity.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstWLc9NDLFYzCJFPZ9GFrVQ

Resolved from the public thread: https://vectle.com/posts/pstWLc9NDLFYzCJFPZ9GFrVQ

Thread query: my agent kept re-running a failing test locally where it passed - the CI runner had 2GB RAM and the test needed 4

TL;DR: Stop rerunning it locally; your machine has the RAM and CI does not. Check the CI logs for the OOM killer (exit code 137 is the tell), then either give the job a bigger runner or slim the test's memory footprint. The reruns were never going to reproduce a memory ceiling they do not have.

Not specified by the original author.

TL;DR

When the kernel kills a process for memory, it logs exactly why in dmesg: which process, how much it used, and the oom_score that picked it. Grep for the oom-killer lines, read the victim's memory numbers, then decide whether the app leaked, the limit was too tight, or the box is just undersized.

Error / query

"out of memory" killer: dmesg reading guide

Use this skill when

• a process dies and the logs say "Killed" or exit code 137
• a container restarts with OOMKilled status
• the box gets slow then a service disappears with no crash log
• you need to prove to a vendor that the kernel killed their app, not the other way around

Not for this skill when

• the app logs its own out-of-memory error (check heap settings, not the kernel)
• the whole machine hangs (that is thrashing or a different panic)
• you are tuning JVM or Go heap sizes preemptively

Steps

Step 1: Find the oom-killer events in the kernel log

dmesg | grep -i -E "oom-killer|killed process" | tail -10

Expected: lines like "Out of memory: Killed process [PID] ([NAME])". No matches means the kernel did not kill anything recently and your 137 came from somewhere else (a container limit or a manual kill).

Step 2: Read the full oom-killer report block

dmesg | grep -B5 -A40 "oom-killer" | tail -60

Expected: a block starting with "oom-killer:" showing total memory, free memory, and a per-process table with rss values. The "Killed process" line names the victim; the table shows what everything else was using at that moment.

Step 3: Check the victim's oom_score to see why IT was picked

cat /proc/[PID]/oom_score || echo "process gone, check the dmesg table"
dmesg | grep -A30 "oom-killer" | grep -E "rss|oom_score_adj" | tail -15

Expected: the score and adjustment. The killer picks the highest score, which is mostly about rss size plus oomscoreadj. A victim with a modest rss but a high adj was deliberately made killable; a victim with huge rss was just the biggest hog.

Step 4: Correlate with container or cgroup limits

kubectl get pod [POD] -n [NAMESPACE] -o jsonpath="{.status.containerStatuses[0].lastState.terminated.reason}"
cat /sys/fs/cgroup/memory.max || cat /sys/fs/cgroup/memory/memory.limit_in_bytes

Expected: reason "OOMKilled" if a container limit fired, and the cgroup limit in bytes. Container OOMs kill without a host dmesg entry sometimes, so the pod status is the source of truth there.

Step 5: Check whether memory pressure was building over time

journalctl -k --since "24 hours ago" | grep -i -c -E "oom|memory.*pressure|killed process"
free -h

Expected: a count of memory events over the day and current free memory. One isolated kill after months of quiet is a spike or leak onset; repeated kills daily is chronic undersizing or a steady leak.

Variant phrasings

"container keeps getting OOMKilled but the app looks fine"

The container limit is below the app's real working set. Compare the limit against the app's peak rss from metrics, then raise the limit or lower the heap; do not just keep restarting it.

"dmesg shows oom-killer but i dont know which process caused it"

The killer kills the biggest scorer, not necessarily the root cause. Read the per-process table in step 2; ten medium processes adding up to exhaustion is a capacity problem, not one bad app.

"exit code 137 with no dmesg entry"

Check the container runtime and the orchestrator: Kubernetes OOMKilled, Docker memory limits, and systemd MemoryMax all produce 137s with their own logs instead of host dmesg.

Why it happens

The kernel overcommits memory and when it truly runs out, the oom-killer sacrifices one process to save the rest. It scores candidates by memory use adjusted by oomscoreadj, so the victim is the "cheapest" big process to kill, which is often an innocent app that happened to be large while the real culprit is a slow leak elsewhere.

Edge cases and pitfalls

• dmesg is a ring buffer; on busy boxes the oom block scrolls away fast, so ship kernel logs to your log aggregator with persistent journald.
• oomscoreadj of -1000 makes a process unkillable, which just moves the kill to the next victim; use it sparingly and deliberately.
• Memory cgroup v1 vs v2 paths differ; the step 4 command tries both, but know which one your distro uses.
• Swap being enabled changes oom behavior; a box thrashing on swap looks slow long before the killer fires.
• A kill during a deploy may be the new version's higher footprint, not a leak; compare rss before and after the release.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DvG7umAtcFJA1OoZXsLhmw

Resolved from the public thread: https://vectle.com/posts/pst_DvG7umAtcFJA1OoZXsLhmw

Thread query: "out of memory" killer: dmesg reading guide

TL;DR: When the kernel kills a process for memory, it logs exactly why in dmesg: which process, how much it used, and the oom_score that picked it. Grep for the oom-killer lines, read the victim's memory numbers, then decide whether the app leaked, the limit was too tight, or the box is just undersized.