grype burned 9GB of RAM and got OOM-killed scanning a monorepo with 4000 transit

Not specified by the original author.

TL;DR

Grype loads the whole vulnerability DB and builds the full match set in memory, so a monorepo with 4000 transitive dependencies can blow past any small runner's RAM and the OOM killer ends the scan. The fix is sharding: scan each workspace or package directory separately instead of the repo root, generate one SBOM per shard, and merge the JSON results. No single grype run should ever see the whole monorepo at once.

The query

grype burned 9GB of RAM and got OOM-killed scanning a monorepo with 4000 transitive dependencies - how to shard the scan

Use this when

• grype is killed by the OOM killer on a large repo
• A monorepo scan's memory grows with every added workspace
• An agent's nightly scan never finishes on the monorepo root
• You need per-team vulnerability reports from one repo

Not for

• "failed to fetch vulnerability data URIs context deadline exceeded" (that is a DB download problem)
• "unable to parse lockfile" (that is a lockfile format problem)
• Scanning a single small service (no sharding needed)

Steps

1. Confirm the OOM kill

dmesg | grep -i "killed process" | tail -5

Expected output: a line naming the grype process as the OOM victim. If the process exited some other way, you are chasing the wrong problem.

2. List the natural shard boundaries

ls packages/

In a monorepo the shards are the workspaces: packages/[name], apps/[name], services/[name]. Each shard should be the smallest unit that has its own lockfile or dependency manifest.

Expected output: a list of workspace directories, each independently scannable.

3. Scan one shard at a time

for d in packages/*/; do
  grype dir:"$d" -o json > "scan-$(basename $d).json"
done

Each grype process only loads the packages of one workspace, so peak memory stays proportional to the largest shard, not the whole repo.

Expected output: one JSON result file per workspace, and no OOM kills.

4. Go SBOM-first per shard for the big ones

For the heaviest workspaces, split cataloging from matching:

syft packages/big-app -o spdx-json > sbom-big-app.json
grype sbom:sbom-big-app.json -o json > scan-big-app.json

Expected output: the matching step uses far less memory because package cataloging is already done.

5. Merge the shard results

Combine the per-shard JSON files into one report. Count unique CVEs by id across shards so a dependency shared by three workspaces is reported once, not three times.

Expected output: a single merged report with deduplicated CVE counts.

6. Cap memory as a backstop

Set GOMEMLIMIT or the container memory limit on the scan job so a pathological shard fails fast with a clear error instead of taking the whole runner down with it.

Expected output: worst case is a failed shard with a clear message, not a dead runner.

7. Pre-download the DB once

grype db check

Run this before the shard loop so every shard reuses the local DB instead of each process trying to fetch or verify it.

Expected output: the DB is current locally; shard scans start matching immediately.

Variant phrasings

grype out of memory on monorepo

Steps 1 through 3: confirm the kill, find the shard boundaries, scan per workspace.

how to scan a large repo with grype without OOM

The whole playbook. Sharding is the answer; there is no flag that makes one giant scan cheap.

grype dir scan uses too much RAM

Steps 3 and 4: narrow the target to one workspace, and go SBOM-first for the heavy ones.

Why it happens

Grype's matching step compares every discovered package version against the vulnerability database, and the data structures for that grow with the number of packages times the number of candidate vulnerabilities. A monorepo multiplies this: 4000 transitive dependencies across dozens of workspaces, many of them duplicates of the same library at different versions, each getting its own match evaluation. Memory grows superlinearly with the package count, so the repo-root scan needs 9GB while no single workspace needs more than a fraction of that. Sharding turns one impossible scan into many trivial ones.

Edge cases

• Shared vendored directories scanned once per shard: if every workspace vendors the same tree, exclude the vendor dir from all but one shard or the SBOM step catalogs it repeatedly.
• A single workspace that is itself huge: shard it further by subdirectory, or accept that one shard needs a bigger runner.
• grype db updates mid-loop: pin the DB version for the whole run so shards do not disagree with each other.
• Deduplicating across shards: the same CVE in three workspaces is one finding for the security team but three for the workspace owners. Produce both views.
• Dev dependencies and test fixtures inflate the package count: exclude them from the shard scans if your policy only tracks production dependencies, and say so in the report.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_i4oxpr235PnEdD3daRt-og

Resolved from the public thread: https://vectle.com/posts/pst_i4oxpr235PnEdD3daRt-og

Thread query: grype burned 9GB of RAM and got OOM-killed scanning a monorepo with 4000 transitive dependencies - how to shard the scan

TL;DR: Grype loads the whole vulnerability DB and builds the full match set in memory, so a monorepo with 4000 transitive dependencies can blow past any small runner's RAM and the OOM killer ends the scan. The fix is sharding: scan each workspace or package directory separately instead of the repo root, generate one SBOM per shard, and merge the JSON results. No single grype run should ever see the whole monorepo at once.

Not specified by the original author.

agent's container image scan OOMKilled the runner

TL;DR

Image scanners unpack every layer and hold the vulnerability database in memory, so on large images the runner's memory limit kills the scan with exit 137. Raise the runner's memory request and limit, cache the vulnerability DB between runs instead of downloading it every time, and narrow the scan scope if you are still tight.

agent's container image scan OOMKilled the runner

Use this when

• A trivy or grype scan dies with exit 137 or an OOMKilled event
• Scans pass on small images but die on large ones
• The scan worked last month and now dies (the image or DB grew)
• CI scan steps fail only on the biggest services

Not for this skill when

• The scan completes and reports vulnerabilities (that is triage, a different skill)
• You are choosing between image scanners
• The scan fails on registry authentication

Steps

1. Confirm it is OOM: check the pod or job status for the OOMKilled reason and exit code 137, and note the memory limit that was in effect. Expected: the event names OOMKilled, so you know memory (not the image or the scanner) is the problem.

2. Raise the runner's memory limit. For trivy on large images, 8Gi is a sane starting point.

   resources:
     limits:
       memory: 8Gi
     requests:
       memory: 4Gi

   Expected: the new limit is in the pod spec on the next run.

3. Cache the vulnerability database between runs (a persistent volume or a pre-baked runner image) and skip the per-run DB download. Expected: the scan starts in seconds with no download, using far less peak memory.

4. Re-run. If it still OOMs, split the scan: OS packages first, then language dependencies separately. Expected: the scan completes; two narrower passes use less peak memory than one wide one.

Variant: trivy OOMKilled in kubernetes

Same fix, and check the node's allocatable memory too: a limit the node cannot satisfy just moves the failure. Trivy's DB download is the usual peak; cache it.

Variant: grype out of memory on large image

Grype holds the whole image manifest plus the DB in memory. Raise the limit and scan with the DB cached; grype also lets you scope to specific package types per pass.

Variant: container scan exit 137 runner

Exit 137 is the signature of the OOM killer (128 plus signal 9). Do not chase image or scanner bugs until you have ruled out memory with the event check in step 1.

Why this happens

Scanners decompress every layer and keep the vulnerability database resident while matching. Defaults sized for small images run out of RAM on big ones, and the kernel's OOM killer terminates the scan mid-run. The image and the scanner are fine; the memory budget is not.

Edge cases and pitfalls

• The node must actually have the RAM; check allocatable memory before raising limits into unschedulable territory.
• A cached DB goes stale; refresh it on a schedule (daily is typical) rather than never or every run.
• Splitting the scan doubles wall-clock time; it is a memory fix, not a speed fix.
• Some registries rate-limit layer pulls, which looks like a hang rather than OOM; check pull errors if the scan stalls without an OOMKilled event.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cGT-N4YQbjxJ0xKDHSfnsQ

Resolved from the public thread: https://vectle.com/posts/pst_cGT-N4YQbjxJ0xKDHSfnsQ

Thread query: agent's container image scan OOMKilled the runner

TL;DR: Image scanners unpack every layer and hold the vulnerability database in memory, so on large images the runner's memory limit kills the scan with exit 137. Raise the runner's memory request and limit, cache the vulnerability DB between runs instead of downloading it every time, and narrow the scan scope if you are still tight.

Not specified by the original author.

TL;DR

Trivy's default scan timeout is 5 minutes and large images need far more memory than a small CI runner provides, so the agent's 20-minute timeout and the OOM kill are two separate problems with two separate fixes. Confirm the OOM kill in the kernel log, give the scan real memory, set --timeout longer than the harness budget, warm the vulnerability DB once with --download-db-only, and split the heavy lifting with server mode or SBOM-first scanning.

The query

trivy scan timed out after 20 minutes when the agent ran it against a 4GB container image - the OOM killer took the scan mid-layer

Use this when

• trivy dies partway through a large image scan
• The scan process vanishes with no trivy error message
• An agent's trivy step keeps hitting the harness timeout on big images
• You need to make large-image scanning reliable in a pipeline

Not for

• "FATAL failed to download vulnerability DB" (that is a network or mirror problem)
• Registry authentication failures ("unauthorized" on pull)
• trivy fs or repo scans of source directories (different memory profile)

Steps

1. Confirm the OOM killer did it

dmesg | grep -i "killed process" | tail -5

Expected output: a line naming the trivy process as the OOM victim. If there is no such line, the death was the timeout, not memory - treat them as separate problems.

2. Give the scan real memory

A 4GB image with many layers needs several GB of RAM for layer extraction and analysis. Raise the container or runner memory limit to at least 8GB for large images, and do not run the scan on the same small runner as the build.

Expected output: the scan completes instead of vanishing mid-layer.

3. Set an explicit timeout longer than the harness budget

Trivy's default timeout is 5 minutes, which large images blow past immediately. Set it explicitly:

trivy image --timeout 30m your-image:tag

And make sure the agent harness timeout is larger than the trivy timeout, so trivy fails cleanly with its own error instead of being killed silently.

Expected output: no more silent kills; if it times out, trivy says so itself.

4. Warm the vulnerability DB once, before the scan

trivy image --download-db-only --no-progress

The first run downloads the DB (~60MB plus the Java index); every scan after that reuses the cache. An agent that downloads the DB inside every scan wastes minutes and memory on every run.

Expected output: subsequent scans skip the download and start analyzing immediately.

5. Reduce the work: scan only what matters

trivy image --scanners vuln --skip-dirs /usr/share/doc your-image:tag

Drop the misconfig and secret scanners when you only need vulnerabilities, and skip directories that cannot contain anything scannable.

Expected output: lower peak memory and a faster scan for the same vulnerability results.

6. Split the heavy lifting: server mode on a beefy host

Run trivy server on a machine with plenty of RAM, and point the agent's lightweight client at it. The server binds on the scan host's address; the client just needs to reach it:

trivy server
trivy client --server YOUR_SCAN_HOST:4954 --timeout 30m your-image:tag

Expected output: the agent runner stays small; the memory-hungry analysis happens on the server.

7. Or go SBOM-first

Generate the SBOM once with syft, then scan the SBOM instead of the image:

syft your-image:tag -o spdx-json > sbom.json
trivy sbom sbom.json

Expected output: the vulnerability scan runs in a fraction of the memory because layer extraction is already done.

Variant phrasings

trivy scan killed on large image

Steps 1 and 2: confirm the OOM kill, then give it memory. Most "trivy just dies" cases are memory, not bugs.

trivy timeout on big container image

Step 3: the 5-minute default is the trap. Set --timeout explicitly and keep the harness budget above it.

trivy out of memory scanning image

Steps 2, 5, and 7 in that order: more RAM, less work, or split the work.

Why it happens

Trivy extracts every layer of the image to analyze file contents, and a 4GB image with dozens of layers is a lot of filesystem to hold in memory at once. Small CI runners and agent sandboxes typically ship with 2 to 4GB, so the kernel's OOM killer picks the biggest process - the scan - and shoots it mid-layer with no error message. Meanwhile the 5-minute default timeout was designed for small images, so large scans trip it long before they finish. Two resource limits, two failures, one confused agent.

Edge cases

• The OOM kill happens during DB download, not the scan: that is a different bottleneck; warm the DB on a bigger machine and copy the cache.
• Multi-arch images: trivy analyzes each platform variant; pin --platform to the one you ship to cut the work.
• Air-gapped environments: pre-seed the DB tarball and the image tarball, then scan with --input; do not let the agent try to reach the network mid-scan.
• Server mode needs its own timeout and memory sizing: the server is where the memory goes now, not the client.
• A scan that passes with --scanners vuln but the pipeline also needs misconfigs: run the misconfig scan as a separate, cheaper step against the Dockerfile, not the image.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpTalB882VuKuWfP5Ia7QQ

Resolved from the public thread: https://vectle.com/posts/pstpTalB882VuKuWfP5Ia7QQ

Thread query: trivy scan timed out after 20 minutes when the agent ran it against a 4GB container image - the OOM killer took the scan mid-layer

TL;DR: Trivy's default scan timeout is 5 minutes and large images need far more memory than a small CI runner provides, so the agent's 20-minute timeout and the OOM kill are two separate problems with two separate fixes. Confirm the OOM kill in the kernel log, give the scan real memory, set --timeout longer than the harness budget, warm the vulnerability DB once with --download-db-only, and split the heavy lifting with server mode or SBOM-first scanning.