the agent's trivy server-mode scan hung for 45 minutes on "analyzing layer 12/31" with no progress output

TL;DR

"Analyzing layer" with no progress output for 45 minutes means the analyzer thread is stuck pulling or decompressing that layer, not that trivy is scanning slowly. Check disk space and memory on the scanner host first, then time the layer pull directly with the container runtime. If the layer is just enormous, scan a smaller artifact (the filesystem or an SBOM) instead of the full image.

The query

the agent's trivy server-mode scan hung for 45 minutes on "analyzing layer 12/31" with no progress output

Use this when

• A trivy server-mode scan stalls on an "analyzing layer" line with no output for tens of minutes.
• The agent cannot tell whether the scan is progressing or dead.
• The image is large (multi-GB, ML images, monorepo build images).

Not for

• Trivy client-mode scans (different code path, different fixes).
• "FATAL failed to download vulnerability DB" (that is a DB mirror problem).
• Scans that complete but report zero vulnerabilities (that is a coverage question).

Steps

Step 1: Confirm the process is alive and see what it is doing

ps aux | grep "[t]rivy" | head -5
df -h "[trivy cache dir]"

Expected output: the trivy process is still running. If CPU is near zero and disk I/O is near zero, it is stuck, not slow. If the cache disk is full, trivy cannot unpack the layer, which is the most common silent stall.

Step 2: Re-run with debug output to see the stuck layer

trivy image --server "[server url]" --debug "[image]" | tail -30

Expected output: debug logs show which layer digest it is stuck on and what operation (download, extract, analyze) is in flight. A stuck download points at the registry or network. A stuck extract points at disk or memory.

Step 3: Time the layer pull directly

time docker pull "[image]"

Expected output: if the pull itself takes tens of minutes or stalls, the problem is the registry or the network path, not trivy. Fix the pull (registry mirror, closer mirror, smaller base image) and the scan unblocks.

Step 4: Check the layer size

docker manifest inspect "[image]" | grep -A 3 "[stuck digest prefix]"

Expected output: the size of the stuck layer. Layers over a few GB (common in ML images with model weights) can take trivy 30+ minutes to decompress and analyze on a small scanner host. That is expected behavior on undersized hardware, not a bug.

Step 5: Scan a smaller artifact instead of the full image

trivy fs --server "[server url]" --scanners vuln "[project dir]"

Expected output: the filesystem scan completes in minutes because it skips layer unpacking entirely. For CI gating, scanning the build context or an SBOM is usually sufficient and far faster than scanning a 4GB image.

Step 6: Set a hard timeout so the agent never waits 45 minutes again

trivy image --server "[server url]" --timeout 20m "[image]"

Expected output: the scan either finishes inside 20 minutes or fails loudly with a timeout error the agent can act on. Pair this with a retry that falls back to the filesystem scan from step 5.

Variant phrasings

trivy server scan stuck on analyzing layer with no output

Same stuck-layer diagnosis. Steps 1-2 identify whether it is disk, network, or layer size.

trivy scan of a 4GB image never finishes in CI

The image is too large for the scanner host. Step 5 (fs or SBOM scan) is the practical fix; step 6 bounds the damage.

trivy hangs after "analyzing" with no progress bar movement

Trivy only advances the progress display per layer, not per megabyte. A single huge layer looks exactly like a hang. Step 4 confirms.

Why it happens

Trivy unpacks each image layer to disk and analyzes it before moving to the next, and the progress display only advances per layer. A single multi-GB layer therefore produces a long silent stretch that is indistinguishable from a hang. On top of that, a full cache disk or a throttled registry makes the stall real. The agent sees one symptom ("stuck on layer 12/31") for three different causes (disk full, slow network, giant layer), which is why step 1 checks the host before blaming trivy.

Edge cases

• In server mode the scan runs partly on the server: check disk and memory on the trivy server host, not just the client.
• A layer that downloads fine but never finishes extracting usually means the cache disk filled mid-extract. Clear old cache with trivy image --clear-cache is too blunt; set --cache-dir on a larger volume instead.
• Registry rate limiting can look like a stall: the download crawls at KB/s with no error. Time the pull in step 3 to catch it.
• Scanning :latest while the tag moves can produce digest mismatches mid-scan. Pin the digest for reproducible scans.
• If the stuck layer is a scratch-built base with no package metadata, trivy finds nothing in it anyway. Excluding it loses no signal.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RxZ50qM5ySp2Rvg5-zdKhw