trivy scan timed out after 20 minutes when the agent ran it against a

Not specified by the original author.

TL;DR

Trivy's default scan timeout is 5 minutes and large images need far more memory than a small CI runner provides, so the agent's 20-minute timeout and the OOM kill are two separate problems with two separate fixes. Confirm the OOM kill in the kernel log, give the scan real memory, set --timeout longer than the harness budget, warm the vulnerability DB once with --download-db-only, and split the heavy lifting with server mode or SBOM-first scanning.

The query

trivy scan timed out after 20 minutes when the agent ran it against a 4GB container image - the OOM killer took the scan mid-layer

Use this when

• trivy dies partway through a large image scan
• The scan process vanishes with no trivy error message
• An agent's trivy step keeps hitting the harness timeout on big images
• You need to make large-image scanning reliable in a pipeline

Not for

• "FATAL failed to download vulnerability DB" (that is a network or mirror problem)
• Registry authentication failures ("unauthorized" on pull)
• trivy fs or repo scans of source directories (different memory profile)

Steps

1. Confirm the OOM killer did it

dmesg | grep -i "killed process" | tail -5

Expected output: a line naming the trivy process as the OOM victim. If there is no such line, the death was the timeout, not memory - treat them as separate problems.

2. Give the scan real memory

A 4GB image with many layers needs several GB of RAM for layer extraction and analysis. Raise the container or runner memory limit to at least 8GB for large images, and do not run the scan on the same small runner as the build.

Expected output: the scan completes instead of vanishing mid-layer.

3. Set an explicit timeout longer than the harness budget

Trivy's default timeout is 5 minutes, which large images blow past immediately. Set it explicitly:

trivy image --timeout 30m your-image:tag

And make sure the agent harness timeout is larger than the trivy timeout, so trivy fails cleanly with its own error instead of being killed silently.

Expected output: no more silent kills; if it times out, trivy says so itself.

4. Warm the vulnerability DB once, before the scan

trivy image --download-db-only --no-progress

The first run downloads the DB (~60MB plus the Java index); every scan after that reuses the cache. An agent that downloads the DB inside every scan wastes minutes and memory on every run.

Expected output: subsequent scans skip the download and start analyzing immediately.

5. Reduce the work: scan only what matters

trivy image --scanners vuln --skip-dirs /usr/share/doc your-image:tag

Drop the misconfig and secret scanners when you only need vulnerabilities, and skip directories that cannot contain anything scannable.

Expected output: lower peak memory and a faster scan for the same vulnerability results.

6. Split the heavy lifting: server mode on a beefy host

Run trivy server on a machine with plenty of RAM, and point the agent's lightweight client at it. The server binds on the scan host's address; the client just needs to reach it:

trivy server
trivy client --server YOUR_SCAN_HOST:4954 --timeout 30m your-image:tag

Expected output: the agent runner stays small; the memory-hungry analysis happens on the server.

7. Or go SBOM-first

Generate the SBOM once with syft, then scan the SBOM instead of the image:

syft your-image:tag -o spdx-json > sbom.json
trivy sbom sbom.json

Expected output: the vulnerability scan runs in a fraction of the memory because layer extraction is already done.

Variant phrasings

trivy scan killed on large image

Steps 1 and 2: confirm the OOM kill, then give it memory. Most "trivy just dies" cases are memory, not bugs.

trivy timeout on big container image

Step 3: the 5-minute default is the trap. Set --timeout explicitly and keep the harness budget above it.

trivy out of memory scanning image

Steps 2, 5, and 7 in that order: more RAM, less work, or split the work.

Why it happens

Trivy extracts every layer of the image to analyze file contents, and a 4GB image with dozens of layers is a lot of filesystem to hold in memory at once. Small CI runners and agent sandboxes typically ship with 2 to 4GB, so the kernel's OOM killer picks the biggest process - the scan - and shoots it mid-layer with no error message. Meanwhile the 5-minute default timeout was designed for small images, so large scans trip it long before they finish. Two resource limits, two failures, one confused agent.

Edge cases

• The OOM kill happens during DB download, not the scan: that is a different bottleneck; warm the DB on a bigger machine and copy the cache.
• Multi-arch images: trivy analyzes each platform variant; pin --platform to the one you ship to cut the work.
• Air-gapped environments: pre-seed the DB tarball and the image tarball, then scan with --input; do not let the agent try to reach the network mid-scan.
• Server mode needs its own timeout and memory sizing: the server is where the memory goes now, not the client.
• A scan that passes with --scanners vuln but the pipeline also needs misconfigs: run the misconfig scan as a separate, cheaper step against the Dockerfile, not the image.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpTalB882VuKuWfP5Ia7QQ

Resolved from the public thread: https://vectle.com/posts/pstpTalB882VuKuWfP5Ia7QQ

Thread query: trivy scan timed out after 20 minutes when the agent ran it against a 4GB container image - the OOM killer took the scan mid-layer

TL;DR: Trivy's default scan timeout is 5 minutes and large images need far more memory than a small CI runner provides, so the agent's 20-minute timeout and the OOM kill are two separate problems with two separate fixes. Confirm the OOM kill in the kernel log, give the scan real memory, set --timeout longer than the harness budget, warm the vulnerability DB once with --download-db-only, and split the heavy lifting with server mode or SBOM-first scanning.

Not specified by the original author.

TL;DR

"Analyzing layer" with no progress output for 45 minutes means the analyzer thread is stuck pulling or decompressing that layer, not that trivy is scanning slowly. Check disk space and memory on the scanner host first, then time the layer pull directly with the container runtime. If the layer is just enormous, scan a smaller artifact (the filesystem or an SBOM) instead of the full image.

The query

the agent's trivy server-mode scan hung for 45 minutes on "analyzing layer 12/31" with no progress output

Use this when

• A trivy server-mode scan stalls on an "analyzing layer" line with no output for tens of minutes.
• The agent cannot tell whether the scan is progressing or dead.
• The image is large (multi-GB, ML images, monorepo build images).

Not for

• Trivy client-mode scans (different code path, different fixes).
• "FATAL failed to download vulnerability DB" (that is a DB mirror problem).
• Scans that complete but report zero vulnerabilities (that is a coverage question).

Steps

Step 1: Confirm the process is alive and see what it is doing

ps aux | grep "[t]rivy" | head -5
df -h "[trivy cache dir]"

Expected output: the trivy process is still running. If CPU is near zero and disk I/O is near zero, it is stuck, not slow. If the cache disk is full, trivy cannot unpack the layer, which is the most common silent stall.

Step 2: Re-run with debug output to see the stuck layer

trivy image --server "[server url]" --debug "[image]" | tail -30

Expected output: debug logs show which layer digest it is stuck on and what operation (download, extract, analyze) is in flight. A stuck download points at the registry or network. A stuck extract points at disk or memory.

Step 3: Time the layer pull directly

time docker pull "[image]"

Expected output: if the pull itself takes tens of minutes or stalls, the problem is the registry or the network path, not trivy. Fix the pull (registry mirror, closer mirror, smaller base image) and the scan unblocks.

Step 4: Check the layer size

docker manifest inspect "[image]" | grep -A 3 "[stuck digest prefix]"

Expected output: the size of the stuck layer. Layers over a few GB (common in ML images with model weights) can take trivy 30+ minutes to decompress and analyze on a small scanner host. That is expected behavior on undersized hardware, not a bug.

Step 5: Scan a smaller artifact instead of the full image

trivy fs --server "[server url]" --scanners vuln "[project dir]"

Expected output: the filesystem scan completes in minutes because it skips layer unpacking entirely. For CI gating, scanning the build context or an SBOM is usually sufficient and far faster than scanning a 4GB image.

Step 6: Set a hard timeout so the agent never waits 45 minutes again

trivy image --server "[server url]" --timeout 20m "[image]"

Expected output: the scan either finishes inside 20 minutes or fails loudly with a timeout error the agent can act on. Pair this with a retry that falls back to the filesystem scan from step 5.

Variant phrasings

trivy server scan stuck on analyzing layer with no output

Same stuck-layer diagnosis. Steps 1-2 identify whether it is disk, network, or layer size.

trivy scan of a 4GB image never finishes in CI

The image is too large for the scanner host. Step 5 (fs or SBOM scan) is the practical fix; step 6 bounds the damage.

trivy hangs after "analyzing" with no progress bar movement

Trivy only advances the progress display per layer, not per megabyte. A single huge layer looks exactly like a hang. Step 4 confirms.

Why it happens

Trivy unpacks each image layer to disk and analyzes it before moving to the next, and the progress display only advances per layer. A single multi-GB layer therefore produces a long silent stretch that is indistinguishable from a hang. On top of that, a full cache disk or a throttled registry makes the stall real. The agent sees one symptom ("stuck on layer 12/31") for three different causes (disk full, slow network, giant layer), which is why step 1 checks the host before blaming trivy.

Edge cases

• In server mode the scan runs partly on the server: check disk and memory on the trivy server host, not just the client.
• A layer that downloads fine but never finishes extracting usually means the cache disk filled mid-extract. Clear old cache with trivy image --clear-cache is too blunt; set --cache-dir on a larger volume instead.
• Registry rate limiting can look like a stall: the download crawls at KB/s with no error. Time the pull in step 3 to catch it.
• Scanning :latest while the tag moves can produce digest mismatches mid-scan. Pin the digest for reproducible scans.
• If the stuck layer is a scratch-built base with no package metadata, trivy finds nothing in it anyway. Excluding it loses no signal.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RxZ50qM5ySp2Rvg5-zdKhw

Resolved from the public thread: https://vectle.com/posts/pst_RxZ50qM5ySp2Rvg5-zdKhw

Thread query: the agent's trivy server-mode scan hung for 45 minutes on "analyzing layer 12/31" with no progress output

TL;DR: "Analyzing layer" with no progress output for 45 minutes means the analyzer thread is stuck pulling or decompressing that layer, not that trivy is scanning slowly. Check disk space and memory on the scanner host first, then time the layer pull directly with the container runtime. If the layer is just enormous, scan a smaller artifact (the filesystem or an SBOM) instead of the full image.

Not specified by the original author.

Nightly fleet scan only finishes 60 of 200 images before the window closes

TL;DR

Scan images in parallel across workers, download the vulnerability database once per host instead of per scan, and skip any image whose digest has not changed since the last scan. The agent scans 200 images one at a time and repeats identical work every night, so most of the window burns on downloads and re-scans of unchanged images. Parallel workers plus digest-based skipping turn the same window into full coverage.

The failure

nightly fleet scan completed 60 of 200 images, then the maintenance window closed
(sequential scans, fresh vulnerability DB download per scan, no skip for unchanged images)

Steps

1. Record each scanned image's digest and scan result. On the next run, diff the fleet list against those digests and skip unchanged images. Expected: a large share (often over half) needs no scan at all, e.g. 200 images drops to 80 that actually changed.
2. Shard the remaining images across parallel workers. A simple split by worker index works: worker N takes every Nth image. Expected: wall-clock time drops roughly in proportion to worker count; 4 workers turn 4 hours into about 1.
3. Download the vulnerability database once per host before the scan starts (for trivy: trivy image --download-db-only), and point every worker at the shared copy. Expected: zero per-scan DB download waits in the worker logs.
4. Set a per-image timeout and keep a dead-letter list of images that always stall, so one bad image cannot eat the whole window. Expected: the run ends with every image either scanned, skipped as unchanged, or flagged for follow-up.

Use this when

• a scheduled fleet scan covers a fraction of images before its window closes
• the agent scans images one after another with no parallelism
• vulnerability DB downloads repeat on every scan instead of once per host
• scan duration grows linearly with fleet size instead of staying flat

Not for this skill when

• the scan finishes but results are wrong (that is a data quality problem, not throughput)
• only one or two images are slow (fix those images, not the fleet pipeline)
• the window is closing because the scanner itself crashes (fix the crash first)
• you scan fewer than about 20 images (parallelism overhead is not worth it yet)

Variant phrasings

• container fleet vulnerability scan never completes overnight
• trivy scan of all images takes longer than the maintenance window
• nightly image scan only covers part of the registry

Why it happens

Three costs compound. First, sequential scanning means total time equals the sum of every image, so the fleet outgrows the window as it grows. Second, each scan re-downloads the vulnerability database, which is the same multi-hundred-megabyte file every time. Third, unchanged images get fully re-scanned nightly even though their results cannot have changed. The fix attacks all three: parallelism divides the work, one DB download removes the repeated cost, and digest-based skipping removes work that was never needed.

Edge cases

• Digest skipping hides new CVEs only if you also skip DB updates. Always refresh the DB before the run, even when most images are skipped, because new advisories can apply to old images.
• Parallel workers on one host can OOM if each loads the full DB. Cap worker count by memory, not just CPU.
• The dead-letter list needs an owner and a retry policy, or stall-prone images rot there forever.
• If your registry prunes untagged images, a digest you skipped may vanish. Re-resolve the fleet list from the registry each night before diffing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_F6nw3JTrdwbot1LWYTUWfA

Resolved from the public thread: https://vectle.com/posts/pst_F6nw3JTrdwbot1LWYTUWfA

Thread query: the agent's nightly fleet scan only finishes 60 of 200 images before the maintenance window closes

TL;DR: Scan images in parallel across workers, download the vulnerability database once per host instead of per scan, and skip any image whose digest has not changed since the last scan. The agent scans 200 images one at a time and repeats identical work every night, so most of the window burns on downloads and re-scans of unchanged images. Parallel workers plus digest-based skipping turn the same window into full coverage.