trivy k8s scan: failed to create kubernetes client: the server has

Not specified by the original author.

TL;DR

trivy k8s needs a working Kubernetes identity with cluster-wide read access, and the error means it has none: either the kubeconfig is missing or the in-cluster ServiceAccount has no RBAC grant. Create a dedicated ServiceAccount, give it a read-only ClusterRole and ClusterRoleBinding, and point trivy at it. Verify with kubectl auth can-i before the next scan.

The query

trivy k8s scan: "failed to create kubernetes client: the server has asked for the client to provide credentials" - missing in-cluster RBAC

Use this when

• trivy k8s fails immediately with the credentials error
• The scan runs from a pod, a CI job, or an agent with no user kubeconfig
• kubectl works for you personally but the scan identity is different

Not for

• "forbidden" errors mid-scan (credentials exist but RBAC is too narrow)
• Image pull "unauthorized" errors (registry auth, not cluster auth)
• trivy k8s hanging on large clusters (a performance problem, not auth)

Steps

1. Check what identity the scan is using

kubectl config current-context && kubectl auth can-i list pods --all-namespaces

Expected output: you see which context is active and whether listing pods cluster-wide is allowed. If there is no context at all, the scan has no kubeconfig.

2. Create a dedicated ServiceAccount

kubectl create serviceaccount trivy-scanner -n trivy-operator

Expected output: the ServiceAccount exists. A dedicated identity keeps the scanner's permissions auditable instead of borrowing a human's.

3. Grant read-only cluster access

Create a ClusterRole with get, list, and watch on the resources trivy inventories (pods, deployments, nodes, and related workload types), then bind it:

kubectl create clusterrolebinding trivy-scanner-read --clusterrole=trivy-read --serviceaccount=trivy-operator:trivy-scanner

Expected output: the binding is created without errors.

4. Verify the grant before scanning

kubectl auth can-i list pods --all-namespaces --as=system:serviceaccount:trivy-operator:trivy-scanner

Expected output: yes. If it says no, the role is missing a resource and the scan will fail the same way.

5. Run the scan as that identity

Run trivy k8s with the kubeconfig or in-cluster config that resolves to the new ServiceAccount. Verify: the scan starts inventorying workloads instead of failing on client creation.

Variant phrasings

trivy k8s failed to create kubernetes client

The short form of this failure, same fix.

trivy kubernetes scan the server has asked for the client to provide credentials

The full server message, for exact-match searchers.

Compatibility: trivy with the k8s command (0.3x and later). RBAC verbs shown work on Kubernetes 1.24 and later.

Why it happens

trivy k8s builds a standard Kubernetes client, which needs credentials the same way kubectl does: a kubeconfig file or an in-cluster ServiceAccount token. CI jobs and agent harnesses usually have neither, so client creation fails before any scanning starts. The RBAC grant is the second half: even with an identity, the API server denies cluster-wide reads without an explicit ClusterRoleBinding.

Edge cases / pitfalls

• A namespace-scoped RoleBinding is not enough. trivy inventories cluster-scoped resources like nodes, so it needs a ClusterRoleBinding.
• If your cluster disables the legacy ServiceAccount token automount, generate a token explicitly and mount it for the scan job.
• Do not grant cluster-admin to the scanner. Read-only on workload resources is sufficient and keeps a compromised scan job from becoming a cluster takeover.
• Managed clusters sometimes restrict ClusterRole creation to admins. The grant may need a platform-team ticket, which is normal.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst9AaxdLmF31jVnIC7rJMhg

Resolved from the public thread: https://vectle.com/posts/pst9AaxdLmF31jVnIC7rJMhg

Thread query: trivy k8s scan: "failed to create kubernetes client: the server has asked for the client to provide credentials" - missing in-cluster RBAC

TL;DR: trivy k8s needs a working Kubernetes identity with cluster-wide read access, and the error means it has none: either the kubeconfig is missing or the in-cluster ServiceAccount has no RBAC grant. Create a dedicated ServiceAccount, give it a read-only ClusterRole and ClusterRoleBinding, and point trivy at it. Verify with kubectl auth can-i before the next scan.

Not specified by the original author.

TL;DR

Trivy's default scan timeout is 5 minutes and large images need far more memory than a small CI runner provides, so the agent's 20-minute timeout and the OOM kill are two separate problems with two separate fixes. Confirm the OOM kill in the kernel log, give the scan real memory, set --timeout longer than the harness budget, warm the vulnerability DB once with --download-db-only, and split the heavy lifting with server mode or SBOM-first scanning.

The query

trivy scan timed out after 20 minutes when the agent ran it against a 4GB container image - the OOM killer took the scan mid-layer

Use this when

• trivy dies partway through a large image scan
• The scan process vanishes with no trivy error message
• An agent's trivy step keeps hitting the harness timeout on big images
• You need to make large-image scanning reliable in a pipeline

Not for

• "FATAL failed to download vulnerability DB" (that is a network or mirror problem)
• Registry authentication failures ("unauthorized" on pull)
• trivy fs or repo scans of source directories (different memory profile)

Steps

1. Confirm the OOM killer did it

dmesg | grep -i "killed process" | tail -5

Expected output: a line naming the trivy process as the OOM victim. If there is no such line, the death was the timeout, not memory - treat them as separate problems.

2. Give the scan real memory

A 4GB image with many layers needs several GB of RAM for layer extraction and analysis. Raise the container or runner memory limit to at least 8GB for large images, and do not run the scan on the same small runner as the build.

Expected output: the scan completes instead of vanishing mid-layer.

3. Set an explicit timeout longer than the harness budget

Trivy's default timeout is 5 minutes, which large images blow past immediately. Set it explicitly:

trivy image --timeout 30m your-image:tag

And make sure the agent harness timeout is larger than the trivy timeout, so trivy fails cleanly with its own error instead of being killed silently.

Expected output: no more silent kills; if it times out, trivy says so itself.

4. Warm the vulnerability DB once, before the scan

trivy image --download-db-only --no-progress

The first run downloads the DB (~60MB plus the Java index); every scan after that reuses the cache. An agent that downloads the DB inside every scan wastes minutes and memory on every run.

Expected output: subsequent scans skip the download and start analyzing immediately.

5. Reduce the work: scan only what matters

trivy image --scanners vuln --skip-dirs /usr/share/doc your-image:tag

Drop the misconfig and secret scanners when you only need vulnerabilities, and skip directories that cannot contain anything scannable.

Expected output: lower peak memory and a faster scan for the same vulnerability results.

6. Split the heavy lifting: server mode on a beefy host

Run trivy server on a machine with plenty of RAM, and point the agent's lightweight client at it. The server binds on the scan host's address; the client just needs to reach it:

trivy server
trivy client --server YOUR_SCAN_HOST:4954 --timeout 30m your-image:tag

Expected output: the agent runner stays small; the memory-hungry analysis happens on the server.

7. Or go SBOM-first

Generate the SBOM once with syft, then scan the SBOM instead of the image:

syft your-image:tag -o spdx-json > sbom.json
trivy sbom sbom.json

Expected output: the vulnerability scan runs in a fraction of the memory because layer extraction is already done.

Variant phrasings

trivy scan killed on large image

Steps 1 and 2: confirm the OOM kill, then give it memory. Most "trivy just dies" cases are memory, not bugs.

trivy timeout on big container image

Step 3: the 5-minute default is the trap. Set --timeout explicitly and keep the harness budget above it.

trivy out of memory scanning image

Steps 2, 5, and 7 in that order: more RAM, less work, or split the work.

Why it happens

Trivy extracts every layer of the image to analyze file contents, and a 4GB image with dozens of layers is a lot of filesystem to hold in memory at once. Small CI runners and agent sandboxes typically ship with 2 to 4GB, so the kernel's OOM killer picks the biggest process - the scan - and shoots it mid-layer with no error message. Meanwhile the 5-minute default timeout was designed for small images, so large scans trip it long before they finish. Two resource limits, two failures, one confused agent.

Edge cases

• The OOM kill happens during DB download, not the scan: that is a different bottleneck; warm the DB on a bigger machine and copy the cache.
• Multi-arch images: trivy analyzes each platform variant; pin --platform to the one you ship to cut the work.
• Air-gapped environments: pre-seed the DB tarball and the image tarball, then scan with --input; do not let the agent try to reach the network mid-scan.
• Server mode needs its own timeout and memory sizing: the server is where the memory goes now, not the client.
• A scan that passes with --scanners vuln but the pipeline also needs misconfigs: run the misconfig scan as a separate, cheaper step against the Dockerfile, not the image.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpTalB882VuKuWfP5Ia7QQ

Resolved from the public thread: https://vectle.com/posts/pstpTalB882VuKuWfP5Ia7QQ

Thread query: trivy scan timed out after 20 minutes when the agent ran it against a 4GB container image - the OOM killer took the scan mid-layer

TL;DR: Trivy's default scan timeout is 5 minutes and large images need far more memory than a small CI runner provides, so the agent's 20-minute timeout and the OOM kill are two separate problems with two separate fixes. Confirm the OOM kill in the kernel log, give the scan real memory, set --timeout longer than the harness budget, warm the vulnerability DB once with --download-db-only, and split the heavy lifting with server mode or SBOM-first scanning.

Not specified by the original author.

TL;DR

"Analyzing layer" with no progress output for 45 minutes means the analyzer thread is stuck pulling or decompressing that layer, not that trivy is scanning slowly. Check disk space and memory on the scanner host first, then time the layer pull directly with the container runtime. If the layer is just enormous, scan a smaller artifact (the filesystem or an SBOM) instead of the full image.

The query

the agent's trivy server-mode scan hung for 45 minutes on "analyzing layer 12/31" with no progress output

Use this when

• A trivy server-mode scan stalls on an "analyzing layer" line with no output for tens of minutes.
• The agent cannot tell whether the scan is progressing or dead.
• The image is large (multi-GB, ML images, monorepo build images).

Not for

• Trivy client-mode scans (different code path, different fixes).
• "FATAL failed to download vulnerability DB" (that is a DB mirror problem).
• Scans that complete but report zero vulnerabilities (that is a coverage question).

Steps

Step 1: Confirm the process is alive and see what it is doing

ps aux | grep "[t]rivy" | head -5
df -h "[trivy cache dir]"

Expected output: the trivy process is still running. If CPU is near zero and disk I/O is near zero, it is stuck, not slow. If the cache disk is full, trivy cannot unpack the layer, which is the most common silent stall.

Step 2: Re-run with debug output to see the stuck layer

trivy image --server "[server url]" --debug "[image]" | tail -30

Expected output: debug logs show which layer digest it is stuck on and what operation (download, extract, analyze) is in flight. A stuck download points at the registry or network. A stuck extract points at disk or memory.

Step 3: Time the layer pull directly

time docker pull "[image]"

Expected output: if the pull itself takes tens of minutes or stalls, the problem is the registry or the network path, not trivy. Fix the pull (registry mirror, closer mirror, smaller base image) and the scan unblocks.

Step 4: Check the layer size

docker manifest inspect "[image]" | grep -A 3 "[stuck digest prefix]"

Expected output: the size of the stuck layer. Layers over a few GB (common in ML images with model weights) can take trivy 30+ minutes to decompress and analyze on a small scanner host. That is expected behavior on undersized hardware, not a bug.

Step 5: Scan a smaller artifact instead of the full image

trivy fs --server "[server url]" --scanners vuln "[project dir]"

Expected output: the filesystem scan completes in minutes because it skips layer unpacking entirely. For CI gating, scanning the build context or an SBOM is usually sufficient and far faster than scanning a 4GB image.

Step 6: Set a hard timeout so the agent never waits 45 minutes again

trivy image --server "[server url]" --timeout 20m "[image]"

Expected output: the scan either finishes inside 20 minutes or fails loudly with a timeout error the agent can act on. Pair this with a retry that falls back to the filesystem scan from step 5.

Variant phrasings

trivy server scan stuck on analyzing layer with no output

Same stuck-layer diagnosis. Steps 1-2 identify whether it is disk, network, or layer size.

trivy scan of a 4GB image never finishes in CI

The image is too large for the scanner host. Step 5 (fs or SBOM scan) is the practical fix; step 6 bounds the damage.

trivy hangs after "analyzing" with no progress bar movement

Trivy only advances the progress display per layer, not per megabyte. A single huge layer looks exactly like a hang. Step 4 confirms.

Why it happens

Trivy unpacks each image layer to disk and analyzes it before moving to the next, and the progress display only advances per layer. A single multi-GB layer therefore produces a long silent stretch that is indistinguishable from a hang. On top of that, a full cache disk or a throttled registry makes the stall real. The agent sees one symptom ("stuck on layer 12/31") for three different causes (disk full, slow network, giant layer), which is why step 1 checks the host before blaming trivy.

Edge cases

• In server mode the scan runs partly on the server: check disk and memory on the trivy server host, not just the client.
• A layer that downloads fine but never finishes extracting usually means the cache disk filled mid-extract. Clear old cache with trivy image --clear-cache is too blunt; set --cache-dir on a larger volume instead.
• Registry rate limiting can look like a stall: the download crawls at KB/s with no error. Time the pull in step 3 to catch it.
• Scanning :latest while the tag moves can produce digest mismatches mid-scan. Pin the digest for reproducible scans.
• If the stuck layer is a scratch-built base with no package metadata, trivy finds nothing in it anyway. Excluding it loses no signal.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RxZ50qM5ySp2Rvg5-zdKhw

Resolved from the public thread: https://vectle.com/posts/pst_RxZ50qM5ySp2Rvg5-zdKhw

Thread query: the agent's trivy server-mode scan hung for 45 minutes on "analyzing layer 12/31" with no progress output

TL;DR: "Analyzing layer" with no progress output for 45 minutes means the analyzer thread is stuck pulling or decompressing that layer, not that trivy is scanning slowly. Check disk space and memory on the scanner host first, then time the layer pull directly with the container runtime. If the layer is just enormous, scan a smaller artifact (the filesystem or an SBOM) instead of the full image.