imagePullSecrets not working": private registry auth debugging

TL;DR

imagePullSecrets fail for mundane reasons: the secret is in the wrong namespace, the .dockerconfigjson is malformed or stale, the secret is not attached to the pod or service account, or the registry URL in the secret does not match the image URL. Verify each link: secret exists, content valid, attached to the pod, URL matches. One broken link, same ImagePullBackOff.

The query

"imagePullSecrets not working": private registry auth debugging

Use this when

• ImagePullBackOff persists despite imagePullSecrets
• Secrets work for one registry but fail for another
• Registry credentials were rotated
• Setting up private registry auth for the first time

Not for when

• Public registry pulls (no auth needed)
• Registry outages (auth is fine, registry is down)
• Image name typos (different error)

Steps

Step 1: Confirm the secret is in the pod's namespace

Get the secret by name in the pod's namespace. Secrets do not cross namespaces; a secret in default does nothing for pods elsewhere. This is the most common miss. Expected output: the secret found in the right namespace, or the namespace mismatch identified.

Step 2: Validate the .dockerconfigjson content

Decode the secret and check the JSON structure: the auths map must contain the exact registry hostname used in the image reference, with valid base64 credentials. Hand-crafted secrets often have hostname mismatches (with vs without https, trailing slashes). Expected output: valid JSON with the registry hostname matching the image URL exactly.

Step 3: Verify the secret is attached to the pod

Check the pod spec's imagePullSecrets list and the service account's imagePullSecrets. A correct secret that is not referenced might as well not exist. Expected output: the secret referenced by the pod or its service account.

Step 4: Test the credentials outside Kubernetes

Use the decoded credentials in a manual docker login and pull. If that fails, the credentials themselves are wrong or expired, and no Kubernetes configuration will fix it. Expected output: credentials proven good (or bad) independent of the cluster.

Step 5: Check credential expiry and rotation

Registry tokens and passwords expire. If the pull worked last month and fails now, the credential rotated. Set up rotation for the secret (external secrets operator or a rotation job) so this does not recur. Expected output: fresh credentials in place, with rotation automated.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst0yUv5RITN4lQ7mkrwBbNw