# agent's container image scan OOMKilled the runner

## TL;DR
Image scanners unpack every layer and hold the vulnerability database in memory, so on large images the runner's memory limit kills the scan with exit 137. Raise the runner's memory request and limit, cache the vulnerability DB between runs instead of downloading it every time, and narrow the scan scope if you are still tight.

```text
agent's container image scan OOMKilled the runner
```

## Use this when
- A trivy or grype scan dies with exit 137 or an OOMKilled event
- Scans pass on small images but die on large ones
- The scan worked last month and now dies (the image or DB grew)
- CI scan steps fail only on the biggest services

## Not for this skill when
- The scan completes and reports vulnerabilities (that is triage, a different skill)
- You are choosing between image scanners
- The scan fails on registry authentication

## Steps
1. Confirm it is OOM: check the pod or job status for the OOMKilled reason and exit code 137, and note the memory limit that was in effect.
   Expected: the event names OOMKilled, so you know memory (not the image or the scanner) is the problem.
2. Raise the runner's memory limit. For trivy on large images, 8Gi is a sane starting point.
   ```yaml
   resources:
     limits:
       memory: 8Gi
     requests:
       memory: 4Gi
   ```
   Expected: the new limit is in the pod spec on the next run.
3. Cache the vulnerability database between runs (a persistent volume or a pre-baked runner image) and skip the per-run DB download.
   Expected: the scan starts in seconds with no download, using far less peak memory.
4. Re-run. If it still OOMs, split the scan: OS packages first, then language dependencies separately.
   Expected: the scan completes; two narrower passes use less peak memory than one wide one.

### Variant: trivy OOMKilled in kubernetes
Same fix, and check the node's allocatable memory too: a limit the node cannot satisfy just moves the failure. Trivy's DB download is the usual peak; cache it.

### Variant: grype out of memory on large image
Grype holds the whole image manifest plus the DB in memory. Raise the limit and scan with the DB cached; grype also lets you scope to specific package types per pass.

### Variant: container scan exit 137 runner
Exit 137 is the signature of the OOM killer (128 plus signal 9). Do not chase image or scanner bugs until you have ruled out memory with the event check in step 1.

## Why this happens
Scanners decompress every layer and keep the vulnerability database resident while matching. Defaults sized for small images run out of RAM on big ones, and the kernel's OOM killer terminates the scan mid-run. The image and the scanner are fine; the memory budget is not.

## Edge cases and pitfalls
- The node must actually have the RAM; check allocatable memory before raising limits into unschedulable territory.
- A cached DB goes stale; refresh it on a schedule (daily is typical) rather than never or every run.
- Splitting the scan doubles wall-clock time; it is a memory fix, not a speed fix.
- Some registries rate-limit layer pulls, which looks like a hang rather than OOM; check pull errors if the scan stalls without an OOMKilled event.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cGT-N4YQbjxJ0xKDHSfnsQ