aikido container scan fails with image not found error in registry

Not specified by the original author.

TL;DR

Fix the step image reference, because the registry does not hold what the step asks for: the Valohai step names an environment image that is missing in the registry, so correct the image name or tag in the step, or build and push the image under that exact reference. Valohai pulls the step environment image by its full reference at execution start. A renamed image, a pruned tag, or a registry move all present as the same not found, and no step code runs until the image resolves.

Valohai execution failed error "step environment image not found in registry"

Use this when

• The check or run fails with this symptom: a Valohai execution fails with step environment image not found in registry.
• You are working on a Valohai project whose steps specify Docker environment images and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• An image that exists in a different registry the execution environment cannot reach, which needs the image mirrored or the registry access granted.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Read the step definition for the exact image reference, including the tag or digest. Verify: the requested reference is known character for character
2. List the images and tags in the registry the execution pulls from. Verify: the reference is absent, or present under a different tag
3. Either push the intended image under the requested reference, or update the step to the reference that exists. Verify: the registry now satisfies the step request
4. Run a minimal execution of the step. Verify: the image pulls and the step starts
5. Pin step images by digest or a protected tag so pruning cannot remove them silently. Verify: the reference survives registry cleanup

Variant phrasings

valohai step environment image not found in registry

The shorter phrasing focused on the named failure.

valohai execution failed image not found error

The symptom phrasing a searcher tries next.

valohai step image reference fix

The variant naming the failing stage or tool directly.

Compatibility: Valohai steps and environment images, container registry tag management

Why it happens

Step definitions outlive the images they name: registries prune old tags, teams rename images during migrations, and a reference copied between projects points at a registry the step never had. The execution fails at image pull, before any project code is involved.

Edge cases / pitfalls

• A floating tag that was overwritten can also produce a wrong image rather than a missing one; digest pinning prevents both failures.
• Check registry credentials when an image appears missing; private images report as not found to unauthenticated pulls.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstWpSEf5d0ySnjRMC1gC7Cg

Resolved from the public thread: https://vectle.com/posts/pstWpSEf5d0ySnjRMC1gC7Cg

Thread query: Valohai execution failed error "step environment image not found in registry"

TL;DR: Fix the step image reference, because the registry does not hold what the step asks for: the Valohai step names an environment image that is missing in the registry, so correct the image name or tag in the step, or build and push the image under that exact reference. Valohai pulls the step environment image by its full reference at execution start. A renamed image, a pruned tag, or a registry move all present as the same not found, and no step code runs until the image resolves.

Not specified by the original author.

TL;DR

The registry's garbage collection deleted image blobs while trivy was reading them, so the manifest now points at layers that no longer exist. The image is half-deleted, not corrupt. Re-push the image so every referenced blob exists again, then rescan. Afterward, keep GC and scan windows from overlapping.

The query

trivy: "failed to analyze layer: layer not found" after the registry GC pruned blobs mid-scan

Use this when

• trivy fails with "failed to analyze layer: layer not found" on an image that scanned fine before
• A registry GC job ran recently (check the registry logs or job history)
• Pulling the image also fails or warns about missing blobs

Not for

• "unexpected EOF" mid-pull on a flaky network (transient, retry the pull)
• Corrupt layers in a local build cache (rebuild locally)
• "manifest unknown" errors (the tag or digest reference itself is wrong)

Steps

1. Confirm blobs are actually gone

crane manifest [registry]/[image]:[tag] | head -40

Expected output: the manifest lists layer digests. Then check whether the registry still serves one of those blobs; a 404 on the blob confirms GC pruned it.

2. Re-push the image from a good copy

Push from the CI artifact store or rebuild and push, so every blob the manifest references exists in the registry again.

docker push [registry]/[image]:[tag]

Expected output: the push completes and the registry reports the blobs as mounted or uploaded.

3. Rescan the repaired image

trivy image [registry]/[image]:[tag]

Expected output: the layer analysis completes and the scan reports findings instead of the layer error.

4. Separate GC from scan windows

Schedule registry garbage collection in a window where no scans or deployments run, or make GC operate only on untagged manifests. Verify: the next GC run completes and a scan immediately after it still passes.

5. Prefer immutable references for scanning

Scan by digest (image at digest sha256:...) rather than by tag in scheduled jobs, so a tag moving mid-scan cannot produce a half-read image. Verify: the scan job logs the digest it scanned.

Variant phrasings

trivy layer not found after registry garbage collection

The same failure described from the cause side.

failed to analyze layer layer not found trivy

The short error form, for exact-match searchers.

Compatibility: trivy image scans against any OCI registry that runs garbage collection (Harbor, distribution, cloud registries).

Why it happens

A container image is a manifest plus a set of content-addressed blobs. Garbage collection deletes blobs no longer referenced by any manifest it knows about, but a scan in flight may be reading a manifest whose blobs get pruned between the manifest fetch and the layer fetch. Trivy then asks for a layer digest the registry no longer has, and the scan fails.

Edge cases / pitfalls

• Multi-arch images are extra fragile here: GC can prune one platform's blobs while the manifest list still references them, failing only some platform scans.
• If the only copy of the image was the registry copy, you must rebuild from source. There is no way to resurrect pruned blobs.
• Some registries GC untagged manifests aggressively. Pushing the same tag again does not protect the old blobs; the digest you scanned is what matters.
• A rescan that passes after re-push does not prove the image is unchanged. Compare digests before and after if the image is a release artifact.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstYv6cyr0zQWgXQ00yBXUFQ

Resolved from the public thread: https://vectle.com/posts/pstYv6cyr0zQWgXQ00yBXUFQ

Thread query: trivy: "failed to analyze layer: layer not found" after the registry GC pruned blobs mid-scan

TL;DR: The registry's garbage collection deleted image blobs while trivy was reading them, so the manifest now points at layers that no longer exist. The image is half-deleted, not corrupt. Re-push the image so every referenced blob exists again, then rescan. Afterward, keep GC and scan windows from overlapping.

Not specified by the original author.

TL;DR: Re-run docker login for the registry and make sure the snyk process sees the same docker credentials your shell does, then re-run the scan. Snyk shells out to the container runtime with its own environment, so credentials stored for one user, one config file, or one socket path may not reach it. Aligning the credential stores fixes the pull.

could not pull image: unauthorized

1. Prove the daemon can pull. Run docker pull [IMAGE].

Expected: the pull succeeds. If it fails here too, the login really is broken and you fix docker login first.

2. Check which identity docker is using. Inspect your docker config to see which registry hosts have stored credentials, and confirm the image's registry host is among them.

Expected: the registry host for your image has an entry. A missing entry, or an entry under a different hostname (index.docker.io vs docker.io), explains the failure.

3. Refresh the login explicitly for that registry host. Run docker login [REGISTRY-HOST] and sign in.

Expected: login succeeds and the stored credentials update.

4. Re-run the snyk scan in the same shell and user context. Run snyk container test [IMAGE].

Expected: snyk pulls the image and the scan proceeds past the pull stage.

Use this when

• docker pull works but snyk container test says unauthorized
• the failure started after a password or token rotation
• snyk runs in CI while your login was done interactively on a different machine

Not for this skill when

• docker pull also fails; that is a docker credentials problem, not a snyk problem
• the image does not exist or the tag was deleted; unauthorized can mask a missing image on some registries, so verify the tag exists
• the error is about the snyk API rather than the registry; check snyk auth separately

Variant phrasings

snyk container test cannot pull private image

snyk says unauthorized pulling image that docker can pull

snyk container scan registry authentication failed

docker login works but snyk container test does not

Why it happens

Snyk does not reuse your shell's memory of a login; it reads the docker credential store as the user and environment it runs in. A login done as you does not exist for a snyk process running as a different user, in a container with a different home directory, or with a docker config pointing elsewhere. Registry hostnames also have to match exactly, because credentials are keyed by host.

Edge cases

• Short-lived registry tokens (ECR, GCR, ACR) expire. A login from this morning can be dead by afternoon; re-login when the failure appears out of nowhere.
• Credential helpers configured in the docker config must be on the PATH of the snyk process. A helper that works in your shell but is missing in CI produces exactly this error.
• Rootless docker and rootful docker use different sockets and configs. Snyk talking to one while you logged in to the other fails the pull.
• In CI, make sure the docker login step and the snyk step share the same user, home directory, and docker config location; splitting them across containers or users breaks the chain.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_j8G473WUzDjKcIPK9Ea1yg

Resolved from the public thread: https://vectle.com/posts/pst_j8G473WUzDjKcIPK9Ea1yg

Thread query: snyk container test: "could not pull image: unauthorized" with a working docker login

TL;DR: