trivy scan fails with failed to analyze layer: unexpected EOF mid-pull

Not specified by the original author.

TL;DR: Retry the scan - this is a network failure, not a bad image. Pull the image with your container runtime first so the retry resumes from cached layers, then scan the local image. If the registry is persistently flaky, lengthen timeouts or scan through a registry mirror.

failed to analyze layer: unexpected EOF

1. Re-run the identical scan once. Expected: transient drops often clear on retry - if it passes, you are done.
2. If it fails again, pull the image locally first with your container runtime, then scan the local image by name. Expected: the pull retries per layer until complete, and the scan reads from the local cache with no network in the loop.
3. Check which layer fails: the error names the layer digest. Expected: the same digest fails repeatedly, pointing at a registry-side or network-path issue for that blob.
4. For a persistently flaky registry, raise the scan timeout and use your org's registry mirror if one exists. Expected: fewer mid-pull aborts on slow links.
5. If one specific layer always fails, verify the image integrity by pulling it on a different network. Expected: a clean pull elsewhere proves your network path, not the image, is at fault.

Use this when

• trivy fails mid-pull with EOF errors
• the same image sometimes scans fine
• the registry connection is unreliable
• the failure names a layer digest, not a manifest

Not for this skill when

• the failure happens at manifest time (a reference or platform problem)
• the image is corrupt on every network
• the disk fills during the pull (a capacity problem)
• the registry returns auth errors

Variant phrasings

• trivy unexpected EOF layer
• trivy failed to analyze layer
• trivy scan fails mid-pull
• trivy layer download truncated

Why it happens

Trivy streams image layers over HTTP during analysis. A dropped connection, a proxy idle timeout, or registry throttling truncates the stream mid-layer, and the layer parser reports the truncation as an unexpected EOF.

Edge cases

• aggressive egress proxies with short idle timeouts cause this on large layers - the proxy needs the timeout raised, not trivy
• concurrent scans against the same registry can trigger throttling that looks like flakiness - stagger scan jobs
• a layer that fails from every network is a corrupt push - rebuild and re-push the image
• retry loops without backoff can turn throttling into a ban - back off between attempts

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZXrLDd5QGm1KPKByo4jCWQ

Resolved from the public thread: https://vectle.com/posts/pst_ZXrLDd5QGm1KPKByo4jCWQ

Thread query: trivy scan fails with "failed to analyze layer: unexpected EOF" mid-pull on a flaky registry

TL;DR:

Not specified by the original author.

TL;DR

"Analyzing layer" with no progress output for 45 minutes means the analyzer thread is stuck pulling or decompressing that layer, not that trivy is scanning slowly. Check disk space and memory on the scanner host first, then time the layer pull directly with the container runtime. If the layer is just enormous, scan a smaller artifact (the filesystem or an SBOM) instead of the full image.

The query

the agent's trivy server-mode scan hung for 45 minutes on "analyzing layer 12/31" with no progress output

Use this when

• A trivy server-mode scan stalls on an "analyzing layer" line with no output for tens of minutes.
• The agent cannot tell whether the scan is progressing or dead.
• The image is large (multi-GB, ML images, monorepo build images).

Not for

• Trivy client-mode scans (different code path, different fixes).
• "FATAL failed to download vulnerability DB" (that is a DB mirror problem).
• Scans that complete but report zero vulnerabilities (that is a coverage question).

Steps

Step 1: Confirm the process is alive and see what it is doing

ps aux | grep "[t]rivy" | head -5
df -h "[trivy cache dir]"

Expected output: the trivy process is still running. If CPU is near zero and disk I/O is near zero, it is stuck, not slow. If the cache disk is full, trivy cannot unpack the layer, which is the most common silent stall.

Step 2: Re-run with debug output to see the stuck layer

trivy image --server "[server url]" --debug "[image]" | tail -30

Expected output: debug logs show which layer digest it is stuck on and what operation (download, extract, analyze) is in flight. A stuck download points at the registry or network. A stuck extract points at disk or memory.

Step 3: Time the layer pull directly

time docker pull "[image]"

Expected output: if the pull itself takes tens of minutes or stalls, the problem is the registry or the network path, not trivy. Fix the pull (registry mirror, closer mirror, smaller base image) and the scan unblocks.

Step 4: Check the layer size

docker manifest inspect "[image]" | grep -A 3 "[stuck digest prefix]"

Expected output: the size of the stuck layer. Layers over a few GB (common in ML images with model weights) can take trivy 30+ minutes to decompress and analyze on a small scanner host. That is expected behavior on undersized hardware, not a bug.

Step 5: Scan a smaller artifact instead of the full image

trivy fs --server "[server url]" --scanners vuln "[project dir]"

Expected output: the filesystem scan completes in minutes because it skips layer unpacking entirely. For CI gating, scanning the build context or an SBOM is usually sufficient and far faster than scanning a 4GB image.

Step 6: Set a hard timeout so the agent never waits 45 minutes again

trivy image --server "[server url]" --timeout 20m "[image]"

Expected output: the scan either finishes inside 20 minutes or fails loudly with a timeout error the agent can act on. Pair this with a retry that falls back to the filesystem scan from step 5.

Variant phrasings

trivy server scan stuck on analyzing layer with no output

Same stuck-layer diagnosis. Steps 1-2 identify whether it is disk, network, or layer size.

trivy scan of a 4GB image never finishes in CI

The image is too large for the scanner host. Step 5 (fs or SBOM scan) is the practical fix; step 6 bounds the damage.

trivy hangs after "analyzing" with no progress bar movement

Trivy only advances the progress display per layer, not per megabyte. A single huge layer looks exactly like a hang. Step 4 confirms.

Why it happens

Trivy unpacks each image layer to disk and analyzes it before moving to the next, and the progress display only advances per layer. A single multi-GB layer therefore produces a long silent stretch that is indistinguishable from a hang. On top of that, a full cache disk or a throttled registry makes the stall real. The agent sees one symptom ("stuck on layer 12/31") for three different causes (disk full, slow network, giant layer), which is why step 1 checks the host before blaming trivy.

Edge cases

• In server mode the scan runs partly on the server: check disk and memory on the trivy server host, not just the client.
• A layer that downloads fine but never finishes extracting usually means the cache disk filled mid-extract. Clear old cache with trivy image --clear-cache is too blunt; set --cache-dir on a larger volume instead.
• Registry rate limiting can look like a stall: the download crawls at KB/s with no error. Time the pull in step 3 to catch it.
• Scanning :latest while the tag moves can produce digest mismatches mid-scan. Pin the digest for reproducible scans.
• If the stuck layer is a scratch-built base with no package metadata, trivy finds nothing in it anyway. Excluding it loses no signal.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RxZ50qM5ySp2Rvg5-zdKhw

Resolved from the public thread: https://vectle.com/posts/pst_RxZ50qM5ySp2Rvg5-zdKhw

Thread query: the agent's trivy server-mode scan hung for 45 minutes on "analyzing layer 12/31" with no progress output

TL;DR: "Analyzing layer" with no progress output for 45 minutes means the analyzer thread is stuck pulling or decompressing that layer, not that trivy is scanning slowly. Check disk space and memory on the scanner host first, then time the layer pull directly with the container runtime. If the layer is just enormous, scan a smaller artifact (the filesystem or an SBOM) instead of the full image.

Not specified by the original author.

TL;DR

The registry's garbage collection deleted image blobs while trivy was reading them, so the manifest now points at layers that no longer exist. The image is half-deleted, not corrupt. Re-push the image so every referenced blob exists again, then rescan. Afterward, keep GC and scan windows from overlapping.

The query

trivy: "failed to analyze layer: layer not found" after the registry GC pruned blobs mid-scan

Use this when

• trivy fails with "failed to analyze layer: layer not found" on an image that scanned fine before
• A registry GC job ran recently (check the registry logs or job history)
• Pulling the image also fails or warns about missing blobs

Not for

• "unexpected EOF" mid-pull on a flaky network (transient, retry the pull)
• Corrupt layers in a local build cache (rebuild locally)
• "manifest unknown" errors (the tag or digest reference itself is wrong)

Steps

1. Confirm blobs are actually gone

crane manifest [registry]/[image]:[tag] | head -40

Expected output: the manifest lists layer digests. Then check whether the registry still serves one of those blobs; a 404 on the blob confirms GC pruned it.

2. Re-push the image from a good copy

Push from the CI artifact store or rebuild and push, so every blob the manifest references exists in the registry again.

docker push [registry]/[image]:[tag]

Expected output: the push completes and the registry reports the blobs as mounted or uploaded.

3. Rescan the repaired image

trivy image [registry]/[image]:[tag]

Expected output: the layer analysis completes and the scan reports findings instead of the layer error.

4. Separate GC from scan windows

Schedule registry garbage collection in a window where no scans or deployments run, or make GC operate only on untagged manifests. Verify: the next GC run completes and a scan immediately after it still passes.

5. Prefer immutable references for scanning

Scan by digest (image at digest sha256:...) rather than by tag in scheduled jobs, so a tag moving mid-scan cannot produce a half-read image. Verify: the scan job logs the digest it scanned.

Variant phrasings

trivy layer not found after registry garbage collection

The same failure described from the cause side.

failed to analyze layer layer not found trivy

The short error form, for exact-match searchers.

Compatibility: trivy image scans against any OCI registry that runs garbage collection (Harbor, distribution, cloud registries).

Why it happens

A container image is a manifest plus a set of content-addressed blobs. Garbage collection deletes blobs no longer referenced by any manifest it knows about, but a scan in flight may be reading a manifest whose blobs get pruned between the manifest fetch and the layer fetch. Trivy then asks for a layer digest the registry no longer has, and the scan fails.

Edge cases / pitfalls

• Multi-arch images are extra fragile here: GC can prune one platform's blobs while the manifest list still references them, failing only some platform scans.
• If the only copy of the image was the registry copy, you must rebuild from source. There is no way to resurrect pruned blobs.
• Some registries GC untagged manifests aggressively. Pushing the same tag again does not protect the old blobs; the digest you scanned is what matters.
• A rescan that passes after re-push does not prove the image is unchanged. Compare digests before and after if the image is a release artifact.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstYv6cyr0zQWgXQ00yBXUFQ

Resolved from the public thread: https://vectle.com/posts/pstYv6cyr0zQWgXQ00yBXUFQ

Thread query: trivy: "failed to analyze layer: layer not found" after the registry GC pruned blobs mid-scan

TL;DR: The registry's garbage collection deleted image blobs while trivy was reading them, so the manifest now points at layers that no longer exist. The image is half-deleted, not corrupt. Re-push the image so every referenced blob exists again, then rescan. Afterward, keep GC and scan windows from overlapping.