ImagePullBackOff with ECR: IRSA token refresh issues

TL;DR

ImagePullBackOff from ECR with IRSA almost always means the pod's service account is not getting valid AWS credentials: the OIDC provider is misconfigured, the IAM role trust policy does not match the service account, or the ECR helper cannot refresh. Check the service account annotation, the role's trust policy, and the exact pull error on the pod, then fix the trust relationship. The image itself is rarely the problem.

Error / query

ImagePullBackOff with ECR: IRSA token refresh issues

Use this skill when

• Pods show ImagePullBackOff pulling from an ECR repository
• The cluster uses IRSA (service account annotated with an IAM role)
• The failure started after a cluster upgrade, addon change, or role edit
• Failed to pull image mentions ECR auth or no basic auth credentials

Not for this skill when

• Pulling from Docker Hub, GHCR, or a non-AWS registry (different auth)
• Using static AWS access keys in a pull secret (check the secret instead)
• The error is manifest unknown or not found (wrong image name/tag)

Steps

Step 1: Read the exact pull error

kubectl describe pod [pod-name] -n [namespace] | grep -A5 "Failed to pull image"

Expected: the real reason, e.g. no basic auth credentials (auth never obtained) or 401 Unauthorized (credentials obtained but rejected). Auth-never-obtained points at IRSA; 401 points at the IAM policy.

Step 2: Verify the service account annotation

kubectl get serviceaccount [sa-name] -n [namespace] -o yaml | grep -i "role-arn"

Expected: an annotation like eks.amazonaws.com/role-arn: arn:aws:iam::[account]:role/[role]. Missing or wrong annotation means the pod never assumes a role; fix the annotation on the service account the pod actually uses.

Step 3: Check the IAM role trust policy

aws iam get-role --role-name [role] --query Role.AssumeRolePolicyDocument

Expected: a trust policy allowing sts:AssumeRoleWithWebIdentity for your cluster's OIDC provider with a StringEquals on the service account (system:serviceaccount:[namespace]:[sa-name]). A mismatch here (wrong namespace, wrong account name) is the single most common cause.

Step 4: Confirm the cluster OIDC provider exists and matches

aws eks describe-cluster --name [cluster] --query "cluster.identity.oidc.issuer"
aws iam list-open-id-connect-providers

Expected: the issuer URL from EKS matches a provider in IAM. After cluster recreations the issuer changes and the old provider entry goes stale; delete the stale one and create the current one.

Step 5: Verify the IAM policy allows ECR pulls

aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::[account]:role/[role] --action-names ecr:BatchGetImage ecr:GetDownloadUrlForLayer ecr:BatchCheckLayerAvailability --resource-arns arn:aws:ecr:[region]:[account]:repository/[repo]

Expected: all three actions allowed. If any is denied, attach the ECR pull permissions (or the managed AmazonEC2ContainerRegistryReadOnly policy) to the role.

Variant phrasings

"eks imagepullbackoff ecr"

Same flow. Steps 2 and 3 fix the majority of cases.

"irsa ecr pull fails"

Focus on the trust policy (step 3) and OIDC provider (step 4); IRSA failures are identity problems, not image problems.

Why it happens

With IRSA, the kubelet gets ECR credentials through a credential chain that depends on the pod's projected service-account token being accepted by AWS STS. Any break in the chain (wrong annotation, trust policy not matching the exact service account, stale OIDC provider, missing ECR permissions) leaves the kubelet with no credentials, and the pull fails before the image is ever contacted.

Edge cases and pitfalls

• Region matters: the ECR repository region must match where the credential helper looks; cross-region pulls need the helper configured for that region.
• Service account changes require pod recreation; editing the SA annotation does not fix already-running pods, only new ones.
• The EKS Pod Identity Agent is a different mechanism from IRSA; if the cluster migrated, check which one is actually configured.
• ImagePullBackOff with a 30+ minute backoff can hide a fix; delete the pod to retry immediately after fixing auth.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vLwkgOLYQlOydszNGIdQTA