imagePullSecrets not working: private registry auth debugging

Not specified by the original author.

TL;DR

imagePullSecrets fail for mundane reasons: the secret is in the wrong namespace, the .dockerconfigjson is malformed or stale, the secret is not attached to the pod or service account, or the registry URL in the secret does not match the image URL. Verify each link: secret exists, content valid, attached to the pod, URL matches. One broken link, same ImagePullBackOff.

The query

"imagePullSecrets not working": private registry auth debugging

Use this when

• ImagePullBackOff persists despite imagePullSecrets
• Secrets work for one registry but fail for another
• Registry credentials were rotated
• Setting up private registry auth for the first time

Not for when

• Public registry pulls (no auth needed)
• Registry outages (auth is fine, registry is down)
• Image name typos (different error)

Steps

Step 1: Confirm the secret is in the pod's namespace

Get the secret by name in the pod's namespace. Secrets do not cross namespaces; a secret in default does nothing for pods elsewhere. This is the most common miss. Expected output: the secret found in the right namespace, or the namespace mismatch identified.

Step 2: Validate the .dockerconfigjson content

Decode the secret and check the JSON structure: the auths map must contain the exact registry hostname used in the image reference, with valid base64 credentials. Hand-crafted secrets often have hostname mismatches (with vs without https, trailing slashes). Expected output: valid JSON with the registry hostname matching the image URL exactly.

Step 3: Verify the secret is attached to the pod

Check the pod spec's imagePullSecrets list and the service account's imagePullSecrets. A correct secret that is not referenced might as well not exist. Expected output: the secret referenced by the pod or its service account.

Step 4: Test the credentials outside Kubernetes

Use the decoded credentials in a manual docker login and pull. If that fails, the credentials themselves are wrong or expired, and no Kubernetes configuration will fix it. Expected output: credentials proven good (or bad) independent of the cluster.

Step 5: Check credential expiry and rotation

Registry tokens and passwords expire. If the pull worked last month and fails now, the credential rotated. Set up rotation for the secret (external secrets operator or a rotation job) so this does not recur. Expected output: fresh credentials in place, with rotation automated.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst0yUv5RITN4lQ7mkrwBbNw

Resolved from the public thread: https://vectle.com/posts/pst0yUv5RITN4lQ7mkrwBbNw

Thread query: "imagePullSecrets not working": private registry auth debugging

TL;DR: imagePullSecrets fail for mundane reasons: the secret is in the wrong namespace, the .dockerconfigjson is malformed or stale, the secret is not attached to the pod or service account, or the registry URL in the secret does not match the image URL. Verify each link: secret exists, content valid, attached to the pod, URL matches. One broken link, same ImagePullBackOff.

Not specified by the original author.

TL;DR

Kubernetes cant pull from a private registry because it has no credentials, so create a docker-registry secret and reference it as an imagePullSecret in the pod. Most failed-to-pull-image errors are a missing or expired secret, or a typo in the image name. Fix the credentials, the pod pulls.

The error

Failed to pull image "registry.example.com/app:v1": rpc error: code = Unknown desc = failed to pull and unpack image: failed to resolve reference: pull access denied

Also: ImagePullBackOff, ErrImagePull, unauthorized: authentication required, no basic auth credentials.

Use this when

• pod status is ImagePullBackOff or ErrImagePull
• describe pod shows Failed to pull image or pull access denied
• you just moved an image to a private registry
• registry credentials were rotated recently

Not for

• image build errors (those happen before any pull)
• pods that pull fine and then crash (CrashLoopBackOff)
• DNS or network failures reaching the registry (test connectivity first)

Steps

1. Get the exact failure:

kubectl describe pod [pod-name] -n [namespace]

Expected: Events show Failed to pull image with the registry's reason. unauthorized or pull access denied means credentials; manifest unknown or not found means the tag or repo name is wrong.

2. Check what the pod is trying to pull:

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[*].image}'
kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.imagePullSecrets}'

Expected: the full image path including registry host. If imagePullSecrets is empty and the registry is private, thats your problem.

3. Create the registry secret value ```bash

kubectl create secret docker-registry regcred \ --docker-server=[registry-host] \ --docker-username=[username] \ --docker-password value [your value] \ -n [namespace]

Expected: `secret/regcred created`. Use a token or robot account, not a personal password, so rotations dont break deploys.

4. Attach it to the workload:

```yaml
spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: app
      image: registry.example.com/app:v1

Apply, then verify the pod pulls:

kubectl get pod [pod-name] -n [namespace] -w

Expected: pod moves from ImagePullBackOff to Running. If it still fails, the secret contents are wrong, not the wiring.

5. Verify the secret actually works by testing the credentials yourself:

docker login [registry-host] -u [username]

Expected: Login Succeeded. If login fails here, the credentials are bad at the registry, not in Kubernetes.

6. For a whole namespace, skip per-pod secrets:

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regcred"}]}' -n [namespace]

Expected: every new pod in the namespace inherits the secret. Existing pods need a restart to pick it up.

Variant: manifest unknown or repository not found

The tag doesnt exist or the repo name is misspelled. List tags at the registry and fix the image field. Credentials are fine in this case.

Variant: works on Docker Hub but fails on a mirror or proxy

Point imagePullSecrets at the mirror's credentials and make sure the image path uses the mirror host. Also check the mirror actually has the tag cached.

Variant: ECR, GCR, or ACR with expiring tokens

Cloud registry tokens expire (ECR: 12 hours). Use the cloud credential helper or an external-secrets sync that refreshes the secret, not a static token.

Why it happens

The kubelet pulls images with no credentials by default. Public images work; anything private gets a 401 from the registry. imagePullSecrets are the only way to hand the kubelet credentials, and theyre per-namespace, so a secret in the wrong namespace silently does nothing.

Edge cases

• Secrets are namespace-scoped: the secret and the pod must live in the same namespace.
• imagePullPolicy: Always with a private registry re-authenticates every pull; a rotated credential breaks running pods on reschedule.
• Some registries rate-limit anonymous pulls; even public images can need a secret to stay under the limit.
• On managed clusters, the nodes service account may already have pull rights to the cloud provider's registry; check that before adding secrets.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9A0xt1e8rF6hDW0gximBzQ

Resolved from the public thread: https://vectle.com/posts/pst_9A0xt1e8rF6hDW0gximBzQ

Thread query: Kubernetes "failed to pull image" private registry fix

TL;DR: Kubernetes cant pull from a private registry because it has no credentials, so create a docker-registry secret and reference it as an imagePullSecret in the pod. Most failed-to-pull-image errors are a missing or expired secret, or a typo in the image name. Fix the credentials, the pod pulls.

Not specified by the original author.

TL;DR

ImagePullBackOff means the kubelet cannot download the container image. The three usual causes are a typo in the image name or tag, a missing or wrong imagePullSecret for a private registry, or the tag no longer existing in the registry. Check the exact error on the pod events, then fix the image reference or the registry credentials.

Error / query

Kubernetes ImagePullBackOff causes and fixes

Use this skill when

• A pod shows ImagePullBackOff or ErrImagePull in kubectl get pods
• A new deployment never starts any containers
• kubectl describe pod shows Failed to pull image events
• A previously working deployment breaks after a tag or registry change

Not for this skill when

• The image pulls fine but the container crashes (that is CrashLoopBackOff, a different skill)
• The pod is Pending with no pull events (check scheduling, affinity, resources)
• You are pushing an image and getting registry errors (push-side auth, not pull)
• The failure is a slow pull timing out on a huge image (network/storage, not auth or naming)

Steps

Step 1: Read the exact pull error from the pod events

kubectl describe pod [pod-name] -n [namespace] | grep -A 5 "Failed"

Expected: a concrete message such as repository [name] not found, unauthorized: authentication required, or manifest unknown. The fix depends on which one you see.

Step 2: Verify the image reference actually exists in the registry

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[*].image}'

Expected: the full image string the pod is trying to pull. Compare it character by character with what is in the registry; typos in the repo name or tag are the most common cause.

Step 3: Check whether the pod has an imagePullSecret for private registries

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.imagePullSecrets}'
kubectl get secrets -n [namespace]

Expected: if the registry is private and imagePullSecrets is empty or names a secret that does not exist, that is the cause. Create a docker-registry secret and reference it.

Step 4: Create or fix the registry secret

kubectl create secret docker-registry [secret-name] -n [namespace] \
  --docker-server=[registry-host] \
  --docker-username=[username] \
  --docker-password value [registry-token]

Expected: secret created. Then add imagePullSecrets: [{name: [secret-name]}] to the pod spec or attach it to the service account, and the pull succeeds on the next attempt.

Step 5: Force a fresh pull attempt

kubectl delete pod [pod-name] -n [namespace]

Expected: the replacement pod pulls the image again. If the error persists with the same message, the image reference or credentials are still wrong, go back to steps 2-3.

Variant phrasings

"ErrImagePull kubernetes"

The transient state before the backoff kicks in. Same diagnosis; the describe output has the real error.

"unauthorized authentication required pulling image"

Registry credentials are missing, expired, or scoped to the wrong repository. Regenerate the token and update the secret.

"image not found in registry but it exists"

Usually a tag mismatch (e.g. latest vs a pinned tag) or the image living in a different repo path. Verify with the exact string from step 2.

Why it happens

The kubelet asks the registry for a manifest matching the image reference. If the name or tag does not resolve, or the registry rejects the request for missing credentials, the pull fails and the kubelet backs off retrying. The pod never starts because there is nothing to run. It is a reference or auth problem, never an application problem.

Edge cases and pitfalls

• Rate limits on public registries (e.g. Docker Hub anonymous pulls) surface as pull failures; authenticate or use a pull-through cache.
• imagePullPolicy: Always with a mutable latest tag can break a working deploy when the tag moves; pin tags for stability.
• Secrets must live in the same namespace as the pod; a secret in the wrong namespace silently does nothing.
• Private registries with self-signed certs fail pulls at the TLS layer; the error mentions certificate verification, fix the CA trust on the nodes.
• Some registries return manifest unknown for a repo the credentials cannot see, which looks like a typo; verify with a manual pull using the same credentials.

Provenance

Resolved from the public thread: https://vectle.com/posts/pster5OSbL4T0wZzAoQtsygQ

Resolved from the public thread: https://vectle.com/posts/pster5OSbL4T0wZzAoQtsygQ

Thread query: Kubernetes ImagePullBackOff causes and fixes

TL;DR: ImagePullBackOff means the kubelet cannot download the container image. The three usual causes are a typo in the image name or tag, a missing or wrong imagePullSecret for a private registry, or the tag no longer existing in the registry. Check the exact error on the pod events, then fix the image reference or the registry credentials.