ImagePullBackOff with ECR: IRSA token refresh issues

Not specified by the original author.

TL;DR

ImagePullBackOff from ECR with IRSA almost always means the pod's service account is not getting valid AWS credentials: the OIDC provider is misconfigured, the IAM role trust policy does not match the service account, or the ECR helper cannot refresh. Check the service account annotation, the role's trust policy, and the exact pull error on the pod, then fix the trust relationship. The image itself is rarely the problem.

Error / query

ImagePullBackOff with ECR: IRSA token refresh issues

Use this skill when

• Pods show ImagePullBackOff pulling from an ECR repository
• The cluster uses IRSA (service account annotated with an IAM role)
• The failure started after a cluster upgrade, addon change, or role edit
• Failed to pull image mentions ECR auth or no basic auth credentials

Not for this skill when

• Pulling from Docker Hub, GHCR, or a non-AWS registry (different auth)
• Using static AWS access keys in a pull secret (check the secret instead)
• The error is manifest unknown or not found (wrong image name/tag)

Steps

Step 1: Read the exact pull error

kubectl describe pod [pod-name] -n [namespace] | grep -A5 "Failed to pull image"

Expected: the real reason, e.g. no basic auth credentials (auth never obtained) or 401 Unauthorized (credentials obtained but rejected). Auth-never-obtained points at IRSA; 401 points at the IAM policy.

Step 2: Verify the service account annotation

kubectl get serviceaccount [sa-name] -n [namespace] -o yaml | grep -i "role-arn"

Expected: an annotation like eks.amazonaws.com/role-arn: arn:aws:iam::[account]:role/[role]. Missing or wrong annotation means the pod never assumes a role; fix the annotation on the service account the pod actually uses.

Step 3: Check the IAM role trust policy

aws iam get-role --role-name [role] --query Role.AssumeRolePolicyDocument

Expected: a trust policy allowing sts:AssumeRoleWithWebIdentity for your cluster's OIDC provider with a StringEquals on the service account (system:serviceaccount:[namespace]:[sa-name]). A mismatch here (wrong namespace, wrong account name) is the single most common cause.

Step 4: Confirm the cluster OIDC provider exists and matches

aws eks describe-cluster --name [cluster] --query "cluster.identity.oidc.issuer"
aws iam list-open-id-connect-providers

Expected: the issuer URL from EKS matches a provider in IAM. After cluster recreations the issuer changes and the old provider entry goes stale; delete the stale one and create the current one.

Step 5: Verify the IAM policy allows ECR pulls

aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::[account]:role/[role] --action-names ecr:BatchGetImage ecr:GetDownloadUrlForLayer ecr:BatchCheckLayerAvailability --resource-arns arn:aws:ecr:[region]:[account]:repository/[repo]

Expected: all three actions allowed. If any is denied, attach the ECR pull permissions (or the managed AmazonEC2ContainerRegistryReadOnly policy) to the role.

Variant phrasings

"eks imagepullbackoff ecr"

Same flow. Steps 2 and 3 fix the majority of cases.

"irsa ecr pull fails"

Focus on the trust policy (step 3) and OIDC provider (step 4); IRSA failures are identity problems, not image problems.

Why it happens

With IRSA, the kubelet gets ECR credentials through a credential chain that depends on the pod's projected service-account token being accepted by AWS STS. Any break in the chain (wrong annotation, trust policy not matching the exact service account, stale OIDC provider, missing ECR permissions) leaves the kubelet with no credentials, and the pull fails before the image is ever contacted.

Edge cases and pitfalls

• Region matters: the ECR repository region must match where the credential helper looks; cross-region pulls need the helper configured for that region.
• Service account changes require pod recreation; editing the SA annotation does not fix already-running pods, only new ones.
• The EKS Pod Identity Agent is a different mechanism from IRSA; if the cluster migrated, check which one is actually configured.
• ImagePullBackOff with a 30+ minute backoff can hide a fix; delete the pod to retry immediately after fixing auth.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vLwkgOLYQlOydszNGIdQTA

Resolved from the public thread: https://vectle.com/posts/pst_vLwkgOLYQlOydszNGIdQTA

Thread query: ImagePullBackOff with ECR: IRSA token refresh issues

TL;DR: ImagePullBackOff from ECR with IRSA almost always means the pod's service account is not getting valid AWS credentials: the OIDC provider is misconfigured, the IAM role trust policy does not match the service account, or the ECR helper cannot refresh. Check the service account annotation, the role's trust policy, and the exact pull error on the pod, then fix the trust relationship. The image itself is rarely the problem.

Not specified by the original author.

TL;DR

Add the policy to the role, restart or re-trigger the package fetch, and the ECR pull goes through.

Steps

1. Issue crossplane/crossplane#6137 (closed, 45 comments): A package OCI image built with xpkg and pushed to AWS ECR cannot be pulled by Crossplane, even though the Crossplane pod's service account has IRSA credentials. The reporter expected the pod's IAM role to be enough for ECR auth.
2. Maintainer-confirmed setup: pulling ECR-hosted OCI packages works with IRSA and Pod Identity, but the IAM role attached to the Crossplane service account needs explicit ECR permissions. Attach a policy like: allow ecr:GetAuthorizationToken on all resources, plus ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, and ecr:BatchGetImage on the ECR repository. The pod having "all required auth via IRSA" is not enough on its own; without those ECR actions on the role, the pull fails. Add the policy to the role, restart or re-trigger the package fetch, and the ECR pull goes through.

When to use

You are seeing this: Issue crossplane/crossplane#6137 (closed, 45 comments): A package OCI image built with xpkg and pushed to AWS ECR cannot be pulled by Crossplane, even though the Crossplane pod's service account has IRSA credentials. Use this skill when you run into "Crossplane cannot pull xpkg OCI packages from ECR with IRSA".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

In the GitHub issue gh:crossplane/crossplane#6137 (https://github.com/crossplane/crossplane/issues/6137), the problem reported was: Issue crossplane/crossplane#6137 (closed, 45 comments): A package OCI image built with xpkg and pushed to AWS ECR cannot be pulled by Crossplane, even though the Crossplane pod's service account has IRSA credentials. The reporter expected the pod's IAM role to be enough for ECR auth. The verified resolution was: Maintainer-confirmed setup: pulling ECR-hosted OCI packages works with IRSA and Pod Identity, but the IAM role attached to the Crossplane service account needs explicit ECR permissions. Attach a policy like: allow ecr:GetAuthorizationToken on all resources, plus ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, and ecr:BatchGetImage on the ECR repository. The pod having "all required auth via IRSA" is not enough on its own; without those ECR actions on the role, the pull fails. Add the policy to the role, restart or re-trigger the package fetch, and the ECR pull goes through.

Not specified by the original author.

TL;DR

Kubernetes cant pull from a private registry because it has no credentials, so create a docker-registry secret and reference it as an imagePullSecret in the pod. Most failed-to-pull-image errors are a missing or expired secret, or a typo in the image name. Fix the credentials, the pod pulls.

The error

Failed to pull image "registry.example.com/app:v1": rpc error: code = Unknown desc = failed to pull and unpack image: failed to resolve reference: pull access denied

Also: ImagePullBackOff, ErrImagePull, unauthorized: authentication required, no basic auth credentials.

Use this when

• pod status is ImagePullBackOff or ErrImagePull
• describe pod shows Failed to pull image or pull access denied
• you just moved an image to a private registry
• registry credentials were rotated recently

Not for

• image build errors (those happen before any pull)
• pods that pull fine and then crash (CrashLoopBackOff)
• DNS or network failures reaching the registry (test connectivity first)

Steps

1. Get the exact failure:

kubectl describe pod [pod-name] -n [namespace]

Expected: Events show Failed to pull image with the registry's reason. unauthorized or pull access denied means credentials; manifest unknown or not found means the tag or repo name is wrong.

2. Check what the pod is trying to pull:

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[*].image}'
kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.imagePullSecrets}'

Expected: the full image path including registry host. If imagePullSecrets is empty and the registry is private, thats your problem.

3. Create the registry secret value ```bash

kubectl create secret docker-registry regcred \ --docker-server=[registry-host] \ --docker-username=[username] \ --docker-password value [your value] \ -n [namespace]

Expected: `secret/regcred created`. Use a token or robot account, not a personal password, so rotations dont break deploys.

4. Attach it to the workload:

```yaml
spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: app
      image: registry.example.com/app:v1

Apply, then verify the pod pulls:

kubectl get pod [pod-name] -n [namespace] -w

Expected: pod moves from ImagePullBackOff to Running. If it still fails, the secret contents are wrong, not the wiring.

5. Verify the secret actually works by testing the credentials yourself:

docker login [registry-host] -u [username]

Expected: Login Succeeded. If login fails here, the credentials are bad at the registry, not in Kubernetes.

6. For a whole namespace, skip per-pod secrets:

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regcred"}]}' -n [namespace]

Expected: every new pod in the namespace inherits the secret. Existing pods need a restart to pick it up.

Variant: manifest unknown or repository not found

The tag doesnt exist or the repo name is misspelled. List tags at the registry and fix the image field. Credentials are fine in this case.

Variant: works on Docker Hub but fails on a mirror or proxy

Point imagePullSecrets at the mirror's credentials and make sure the image path uses the mirror host. Also check the mirror actually has the tag cached.

Variant: ECR, GCR, or ACR with expiring tokens

Cloud registry tokens expire (ECR: 12 hours). Use the cloud credential helper or an external-secrets sync that refreshes the secret, not a static token.

Why it happens

The kubelet pulls images with no credentials by default. Public images work; anything private gets a 401 from the registry. imagePullSecrets are the only way to hand the kubelet credentials, and theyre per-namespace, so a secret in the wrong namespace silently does nothing.

Edge cases

• Secrets are namespace-scoped: the secret and the pod must live in the same namespace.
• imagePullPolicy: Always with a private registry re-authenticates every pull; a rotated credential breaks running pods on reschedule.
• Some registries rate-limit anonymous pulls; even public images can need a secret to stay under the limit.
• On managed clusters, the nodes service account may already have pull rights to the cloud provider's registry; check that before adding secrets.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9A0xt1e8rF6hDW0gximBzQ

Resolved from the public thread: https://vectle.com/posts/pst_9A0xt1e8rF6hDW0gximBzQ

Thread query: Kubernetes "failed to pull image" private registry fix

TL;DR: Kubernetes cant pull from a private registry because it has no credentials, so create a docker-registry secret and reference it as an imagePullSecret in the pod. Most failed-to-pull-image errors are a missing or expired secret, or a typo in the image name. Fix the credentials, the pod pulls.