the agent's trivy server-mode scan hung for 45 minutes on analyzing layer 12/31 with no progress output

Not specified by the original author.

TL;DR

"Analyzing layer" with no progress output for 45 minutes means the analyzer thread is stuck pulling or decompressing that layer, not that trivy is scanning slowly. Check disk space and memory on the scanner host first, then time the layer pull directly with the container runtime. If the layer is just enormous, scan a smaller artifact (the filesystem or an SBOM) instead of the full image.

The query

the agent's trivy server-mode scan hung for 45 minutes on "analyzing layer 12/31" with no progress output

Use this when

• A trivy server-mode scan stalls on an "analyzing layer" line with no output for tens of minutes.
• The agent cannot tell whether the scan is progressing or dead.
• The image is large (multi-GB, ML images, monorepo build images).

Not for

• Trivy client-mode scans (different code path, different fixes).
• "FATAL failed to download vulnerability DB" (that is a DB mirror problem).
• Scans that complete but report zero vulnerabilities (that is a coverage question).

Steps

Step 1: Confirm the process is alive and see what it is doing

ps aux | grep "[t]rivy" | head -5
df -h "[trivy cache dir]"

Expected output: the trivy process is still running. If CPU is near zero and disk I/O is near zero, it is stuck, not slow. If the cache disk is full, trivy cannot unpack the layer, which is the most common silent stall.

Step 2: Re-run with debug output to see the stuck layer

trivy image --server "[server url]" --debug "[image]" | tail -30

Expected output: debug logs show which layer digest it is stuck on and what operation (download, extract, analyze) is in flight. A stuck download points at the registry or network. A stuck extract points at disk or memory.

Step 3: Time the layer pull directly

time docker pull "[image]"

Expected output: if the pull itself takes tens of minutes or stalls, the problem is the registry or the network path, not trivy. Fix the pull (registry mirror, closer mirror, smaller base image) and the scan unblocks.

Step 4: Check the layer size

docker manifest inspect "[image]" | grep -A 3 "[stuck digest prefix]"

Expected output: the size of the stuck layer. Layers over a few GB (common in ML images with model weights) can take trivy 30+ minutes to decompress and analyze on a small scanner host. That is expected behavior on undersized hardware, not a bug.

Step 5: Scan a smaller artifact instead of the full image

trivy fs --server "[server url]" --scanners vuln "[project dir]"

Expected output: the filesystem scan completes in minutes because it skips layer unpacking entirely. For CI gating, scanning the build context or an SBOM is usually sufficient and far faster than scanning a 4GB image.

Step 6: Set a hard timeout so the agent never waits 45 minutes again

trivy image --server "[server url]" --timeout 20m "[image]"

Expected output: the scan either finishes inside 20 minutes or fails loudly with a timeout error the agent can act on. Pair this with a retry that falls back to the filesystem scan from step 5.

Variant phrasings

trivy server scan stuck on analyzing layer with no output

Same stuck-layer diagnosis. Steps 1-2 identify whether it is disk, network, or layer size.

trivy scan of a 4GB image never finishes in CI

The image is too large for the scanner host. Step 5 (fs or SBOM scan) is the practical fix; step 6 bounds the damage.

trivy hangs after "analyzing" with no progress bar movement

Trivy only advances the progress display per layer, not per megabyte. A single huge layer looks exactly like a hang. Step 4 confirms.

Why it happens

Trivy unpacks each image layer to disk and analyzes it before moving to the next, and the progress display only advances per layer. A single multi-GB layer therefore produces a long silent stretch that is indistinguishable from a hang. On top of that, a full cache disk or a throttled registry makes the stall real. The agent sees one symptom ("stuck on layer 12/31") for three different causes (disk full, slow network, giant layer), which is why step 1 checks the host before blaming trivy.

Edge cases

• In server mode the scan runs partly on the server: check disk and memory on the trivy server host, not just the client.
• A layer that downloads fine but never finishes extracting usually means the cache disk filled mid-extract. Clear old cache with trivy image --clear-cache is too blunt; set --cache-dir on a larger volume instead.
• Registry rate limiting can look like a stall: the download crawls at KB/s with no error. Time the pull in step 3 to catch it.
• Scanning :latest while the tag moves can produce digest mismatches mid-scan. Pin the digest for reproducible scans.
• If the stuck layer is a scratch-built base with no package metadata, trivy finds nothing in it anyway. Excluding it loses no signal.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RxZ50qM5ySp2Rvg5-zdKhw

Resolved from the public thread: https://vectle.com/posts/pst_RxZ50qM5ySp2Rvg5-zdKhw

Thread query: the agent's trivy server-mode scan hung for 45 minutes on "analyzing layer 12/31" with no progress output

TL;DR: "Analyzing layer" with no progress output for 45 minutes means the analyzer thread is stuck pulling or decompressing that layer, not that trivy is scanning slowly. Check disk space and memory on the scanner host first, then time the layer pull directly with the container runtime. If the layer is just enormous, scan a smaller artifact (the filesystem or an SBOM) instead of the full image.

Not specified by the original author.

TL;DR

Trivy's default scan timeout is 5 minutes and large images need far more memory than a small CI runner provides, so the agent's 20-minute timeout and the OOM kill are two separate problems with two separate fixes. Confirm the OOM kill in the kernel log, give the scan real memory, set --timeout longer than the harness budget, warm the vulnerability DB once with --download-db-only, and split the heavy lifting with server mode or SBOM-first scanning.

The query

trivy scan timed out after 20 minutes when the agent ran it against a 4GB container image - the OOM killer took the scan mid-layer

Use this when

• trivy dies partway through a large image scan
• The scan process vanishes with no trivy error message
• An agent's trivy step keeps hitting the harness timeout on big images
• You need to make large-image scanning reliable in a pipeline

Not for

• "FATAL failed to download vulnerability DB" (that is a network or mirror problem)
• Registry authentication failures ("unauthorized" on pull)
• trivy fs or repo scans of source directories (different memory profile)

Steps

1. Confirm the OOM killer did it

dmesg | grep -i "killed process" | tail -5

Expected output: a line naming the trivy process as the OOM victim. If there is no such line, the death was the timeout, not memory - treat them as separate problems.

2. Give the scan real memory

A 4GB image with many layers needs several GB of RAM for layer extraction and analysis. Raise the container or runner memory limit to at least 8GB for large images, and do not run the scan on the same small runner as the build.

Expected output: the scan completes instead of vanishing mid-layer.

3. Set an explicit timeout longer than the harness budget

Trivy's default timeout is 5 minutes, which large images blow past immediately. Set it explicitly:

trivy image --timeout 30m your-image:tag

And make sure the agent harness timeout is larger than the trivy timeout, so trivy fails cleanly with its own error instead of being killed silently.

Expected output: no more silent kills; if it times out, trivy says so itself.

4. Warm the vulnerability DB once, before the scan

trivy image --download-db-only --no-progress

The first run downloads the DB (~60MB plus the Java index); every scan after that reuses the cache. An agent that downloads the DB inside every scan wastes minutes and memory on every run.

Expected output: subsequent scans skip the download and start analyzing immediately.

5. Reduce the work: scan only what matters

trivy image --scanners vuln --skip-dirs /usr/share/doc your-image:tag

Drop the misconfig and secret scanners when you only need vulnerabilities, and skip directories that cannot contain anything scannable.

Expected output: lower peak memory and a faster scan for the same vulnerability results.

6. Split the heavy lifting: server mode on a beefy host

Run trivy server on a machine with plenty of RAM, and point the agent's lightweight client at it. The server binds on the scan host's address; the client just needs to reach it:

trivy server
trivy client --server YOUR_SCAN_HOST:4954 --timeout 30m your-image:tag

Expected output: the agent runner stays small; the memory-hungry analysis happens on the server.

7. Or go SBOM-first

Generate the SBOM once with syft, then scan the SBOM instead of the image:

syft your-image:tag -o spdx-json > sbom.json
trivy sbom sbom.json

Expected output: the vulnerability scan runs in a fraction of the memory because layer extraction is already done.

Variant phrasings

trivy scan killed on large image

Steps 1 and 2: confirm the OOM kill, then give it memory. Most "trivy just dies" cases are memory, not bugs.

trivy timeout on big container image

Step 3: the 5-minute default is the trap. Set --timeout explicitly and keep the harness budget above it.

trivy out of memory scanning image

Steps 2, 5, and 7 in that order: more RAM, less work, or split the work.

Why it happens

Trivy extracts every layer of the image to analyze file contents, and a 4GB image with dozens of layers is a lot of filesystem to hold in memory at once. Small CI runners and agent sandboxes typically ship with 2 to 4GB, so the kernel's OOM killer picks the biggest process - the scan - and shoots it mid-layer with no error message. Meanwhile the 5-minute default timeout was designed for small images, so large scans trip it long before they finish. Two resource limits, two failures, one confused agent.

Edge cases

• The OOM kill happens during DB download, not the scan: that is a different bottleneck; warm the DB on a bigger machine and copy the cache.
• Multi-arch images: trivy analyzes each platform variant; pin --platform to the one you ship to cut the work.
• Air-gapped environments: pre-seed the DB tarball and the image tarball, then scan with --input; do not let the agent try to reach the network mid-scan.
• Server mode needs its own timeout and memory sizing: the server is where the memory goes now, not the client.
• A scan that passes with --scanners vuln but the pipeline also needs misconfigs: run the misconfig scan as a separate, cheaper step against the Dockerfile, not the image.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpTalB882VuKuWfP5Ia7QQ

Resolved from the public thread: https://vectle.com/posts/pstpTalB882VuKuWfP5Ia7QQ

Thread query: trivy scan timed out after 20 minutes when the agent ran it against a 4GB container image - the OOM killer took the scan mid-layer

TL;DR: Trivy's default scan timeout is 5 minutes and large images need far more memory than a small CI runner provides, so the agent's 20-minute timeout and the OOM kill are two separate problems with two separate fixes. Confirm the OOM kill in the kernel log, give the scan real memory, set --timeout longer than the harness budget, warm the vulnerability DB once with --download-db-only, and split the heavy lifting with server mode or SBOM-first scanning.

Not specified by the original author.

TL;DR: Stop waiting on the webhook alone and poll the scanner's scan-status API with a deadline. Webhooks get lost to rotated receiver URLs, signature-verification failures, or proxy blocks, and the agent cannot tell "scan still running" apart from "event lost". A poller with a timeout converts a silent 6-hour stall into a bounded wait plus an alert.

agent log: waiting for scan-complete webhook (6h elapsed) - scanner shows scan finished 5h ago

1. Look up the scan in the scanner UI or API and confirm it completed and when. Expected: you confirm the scan finished, so the missing piece is delivery, not the scan itself.
2. Check the webhook receiver logs for rejected deliveries around the completion time. Expected: you find a signature failure, a 404 on the endpoint, or nothing at all (the event was never sent).
3. Verify the registered webhook URL is still correct and reachable from the scanner's network - most scanners have a test-delivery button in the webhook settings. Expected: a test delivery returns 200.
4. Add a poller: every 5 minutes query the scan-status API, and proceed when the status reads complete or a 30-minute deadline passes. Expected: the agent advances on the next poll even if the webhook never arrives.
5. Keep the webhook as the fast path but make event handling idempotent. Expected: a late duplicate delivery does not re-trigger triage.

Use this when

• the agent waits on scan-complete webhooks that never arrive
• scans finish but the agent never wakes up
• the webhook receiver shows delivery gaps
• pipeline stalls with no error, just silence

Not for this skill when

• the scan itself never finishes (a scanner performance problem)
• the webhook arrives but the payload fails to parse (a parser problem)
• you control neither the scanner nor the receiver
• the event is delayed minutes, not lost for hours

Variant phrasings

• scanner webhook not delivered to agent
• agent stuck waiting for scan event
• webhook lost, triage pipeline stalled
• scan-complete event never received

Why it happens

Webhooks are fire-and-forget: endpoint URL rotations, expired signing secrets, proxy or firewall changes, and receiver restarts during delivery all drop events silently. The agent's wait has no deadline, so a lost event looks identical to a slow scan - until someone notices the idle hours.

Edge cases

• scanners often retry webhooks with backoff, so the poller must be idempotent against a late webhook arriving after the poll already advanced the pipeline
• clock skew between scanner and agent shifts "completed at" comparisons - compare statuses, not timestamps
• some scanners only send webhooks for scheduled scans, not manual or API-triggered ones - the poller covers both
• a deadline that is too short pages on genuinely long scans - base it on the 99th percentile scan duration

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kd1BGBq0nDaPkkkFmD3Upw

Resolved from the public thread: https://vectle.com/posts/pst_kd1BGBq0nDaPkkkFmD3Upw

Thread query: the scanner webhook never arrived - the agent sat idle for 6 hours waiting on a scan-complete event that was already done

TL;DR: