agent's container image scan OOMKilled the runner

agent's container image scan OOMKilled the runner

TL;DR

Image scanners unpack every layer and hold the vulnerability database in memory, so on large images the runner's memory limit kills the scan with exit 137. Raise the runner's memory request and limit, cache the vulnerability DB between runs instead of downloading it every time, and narrow the scan scope if you are still tight.

agent's container image scan OOMKilled the runner

Use this when

• A trivy or grype scan dies with exit 137 or an OOMKilled event
• Scans pass on small images but die on large ones
• The scan worked last month and now dies (the image or DB grew)
• CI scan steps fail only on the biggest services

Not for this skill when

• The scan completes and reports vulnerabilities (that is triage, a different skill)
• You are choosing between image scanners
• The scan fails on registry authentication

Steps

1. Confirm it is OOM: check the pod or job status for the OOMKilled reason and exit code 137, and note the memory limit that was in effect. Expected: the event names OOMKilled, so you know memory (not the image or the scanner) is the problem.

2. Raise the runner's memory limit. For trivy on large images, 8Gi is a sane starting point.

   resources:
     limits:
       memory: 8Gi
     requests:
       memory: 4Gi

   Expected: the new limit is in the pod spec on the next run.

3. Cache the vulnerability database between runs (a persistent volume or a pre-baked runner image) and skip the per-run DB download. Expected: the scan starts in seconds with no download, using far less peak memory.

4. Re-run. If it still OOMs, split the scan: OS packages first, then language dependencies separately. Expected: the scan completes; two narrower passes use less peak memory than one wide one.

Variant: trivy OOMKilled in kubernetes

Same fix, and check the node's allocatable memory too: a limit the node cannot satisfy just moves the failure. Trivy's DB download is the usual peak; cache it.

Variant: grype out of memory on large image

Grype holds the whole image manifest plus the DB in memory. Raise the limit and scan with the DB cached; grype also lets you scope to specific package types per pass.

Variant: container scan exit 137 runner

Exit 137 is the signature of the OOM killer (128 plus signal 9). Do not chase image or scanner bugs until you have ruled out memory with the event check in step 1.

Why this happens

Scanners decompress every layer and keep the vulnerability database resident while matching. Defaults sized for small images run out of RAM on big ones, and the kernel's OOM killer terminates the scan mid-run. The image and the scanner are fine; the memory budget is not.

Edge cases and pitfalls

• The node must actually have the RAM; check allocatable memory before raising limits into unschedulable territory.
• A cached DB goes stale; refresh it on a schedule (daily is typical) rather than never or every run.
• Splitting the scan doubles wall-clock time; it is a memory fix, not a speed fix.
• Some registries rate-limit layer pulls, which looks like a hang rather than OOM; check pull errors if the scan stalls without an OOMKilled event.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cGT-N4YQbjxJ0xKDHSfnsQ