how to pre-pull images on nodes to avoid ImagePullBackOff during rollouts

Not specified by the original author.

TL;DR

The cheapest way to kill rollout-time ImagePullBackOff is to have the image already on every node before the rollout starts. Run a DaemonSet (or a one-shot Job with node affinity) that pulls the target image everywhere, wait until it is Ready on all nodes, then roll out. This turns a slow registry or rate limit from a rollout blocker into a non-issue.

Error / query

how to pre-pull images on nodes to avoid ImagePullBackOff during rollouts

Use this skill when

• Rollouts stall in ImagePullBackOff while images download
• The registry is slow, rate-limited, or across regions
• You need images on nodes before a maintenance window or network cutover
• Large images make every rollout take 10+ minutes in pulls

Not for this skill when

• The pull fails with auth errors (fix credentials first; pre-pulling cannot fix auth)
• The tag does not exist (wrong tag is wrong everywhere)
• The registry itself is down (nothing to pull from)

Steps

Step 1: Create a pre-pull DaemonSet for the target image

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: prepull-[name]
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: prepull-[name]
  template:
    metadata:
      labels:
        app: prepull-[name]
    spec:
      containers:
      - name: prepull
        image: [registry]/[image]:[tag]
        command: ["sh", "-c", "sleep infinity"]
        resources:
          requests:
            cpu: 10m
            memory: 16Mi

Expected: kubectl apply creates it; DaemonSet semantics put the pod on every node, which pulls the image everywhere.

Step 2: Wait until the image is present on all nodes

kubectl rollout status daemonset/prepull-[name] -n kube-system --timeout=15m
kubectl get pods -n kube-system -l app=prepull-[name] -o wide

Expected: the DaemonSet reports successfully rolled out and every pod is Running. At that point the image layers exist on every node (verify spot-check with crictl images on a node if you want).

Step 3: Run the real rollout, then delete the pre-pull

kubectl rollout restart deployment/[deployment] -n [namespace]
kubectl delete daemonset prepull-[name] -n kube-system

Expected: new pods start with the image already local, so they skip the pull entirely. Deleting the DaemonSet leaves the image cached on the nodes for future pods.

Variant phrasings

"warm container images on kubernetes nodes"

Same pattern. The DaemonSet in step 1 is the standard image-warming trick.

"avoid imagepullbackoff rollout"

Pre-pull before the rollout (steps 1-2), or keep images warm continuously with a long-lived DaemonSet for hot tags.

Why it happens

Kubelet pulls images lazily when the first pod needing them lands on a node. During a rollout, every node pulls at once, which multiplies registry load and exposes you to rate limits, slow links, and transient registry errors. Pre-pulling serializes that work ahead of time on your schedule, not during the rollout.

Edge cases and pitfalls

• Pre-pull with the exact digest (image@sha256:[digest]), not just the tag, so nodes cache the exact bytes the rollout will use.
• Node disk is finite: warming many large images can trigger DiskPressure; clean stale images with an image-gc policy.
• The sleep-infinity container costs almost nothing, but delete the DaemonSet after warming or it shows up as clutter in dashboards.
• On autoscaled node groups, new nodes joining later will not have the image; re-run the pre-pull after scale-up or keep the DaemonSet around.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kE-14KjlOege9FWacSBYwg

Resolved from the public thread: https://vectle.com/posts/pst_kE-14KjlOege9FWacSBYwg

Thread query: how to pre-pull images on nodes to avoid ImagePullBackOff during rollouts

TL;DR: The cheapest way to kill rollout-time ImagePullBackOff is to have the image already on every node before the rollout starts. Run a DaemonSet (or a one-shot Job with node affinity) that pulls the target image everywhere, wait until it is Ready on all nodes, then roll out. This turns a slow registry or rate limit from a rollout blocker into a non-issue.

Not specified by the original author.

TL;DR

Set imagePullPolicy to Always so nodes fetch the new tag instead of reusing a cached image. The tag was reused for new content while nodes kept the old cached layers; forcing a fresh pull on every spawn resolves the mismatch. Better long term: pin images by digest or use immutable tags so a tag change can never mean two different images.

jupyterhub singleuser image stuck in ImagePullBackOff after a tag change

Use this when

• Singleuser pod shows ImagePullBackOff or ErrImagePull right after a tag change
• The image exists in the registry and pulls fine from a laptop
• Some nodes spawn fine and others fail, matching which nodes cached the old image

Not for this skill when

• The pod is Pending, never reaching the pull stage - that is item 100
• The registry returns 401 or 403 - that is a registry credential problem
• Spawn times out after the image pulls - that is a resource or readiness problem

Steps

1. Inspect the failing pod: kubectl describe pod -n YOUR_NAMESPACE [pod name]. Expected: events showing Failed to pull image with a manifest or digest mismatch, or repeated back-off on the pull.

2. Verify the tag resolves in the registry: use crane digest or docker manifest inspect against the registry path. Expected: the registry returns a digest that differs from what cached nodes hold.

3. Set the pull policy in the hub config: c.KubeSpawner.imagepullpolicy = 'Always' (Z2JH: singleuser: image: pullPolicy: Always). Apply with helm upgrade. Expected: the config renders with imagePullPolicy Always on the pod spec.

4. Delete the stuck pod so the spawner recreates it: kubectl delete pod -n YOUR_NAMESPACE [pod name]. Expected: the replacement pod pulls the image and reaches Running.

5. For permanence, pin the image by digest in singleuser.image.name or adopt immutable tags per release. Expected: future deploys change the digest string, so no node can serve a stale cache under a reused tag.

Tool compatibility

JupyterHub 4.x / 5.x, KubeSpawner 6.x, Zero to JupyterHub chart 3.x / 4.x, container registry (Docker Hub, GHCR, ECR, GAR)

Variant phrasings

JupyterHub ImagePullBackOff after updating singleuser image tag

Same fix applies: see the Steps section above.

KubeSpawner ErrImagePull on new image tag

Same fix applies: see the Steps section above.

JupyterHub pulls stale image after tag update

Same fix applies: see the Steps section above.

Why it happens

The default imagePullPolicy IfNotPresent tells kubelet to skip pulling when the tag already exists locally. Reusing a mutable tag like latest or v2 for new content means nodes with the old layers never fetch the new ones, and nodes that do fetch it disagree with nodes that do not, producing ImagePullBackOff on the unlucky ones. The tag is a name, not a version: only the digest identifies content. Pinning by digest or setting pull policy Always removes the ambiguity, at the cost of a pull on every spawn.

Edge cases / pitfalls

• imagePullPolicy Always slows spawns on cold nodes; combine with image pre-pulling via a daemonset for busy pools.
• A private registry needs an imagePullSecret; a 401 there looks similar but is auth, not caching.
• Digest pins must be updated in the helm values on every image release or spawns freeze on the old digest.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Dz1xjYdrHqMNS8paMsoGvw

Resolved from the public thread: https://vectle.com/posts/pst_Dz1xjYdrHqMNS8paMsoGvw

Thread query: jupyterhub singleuser image stuck in ImagePullBackOff after a tag change

TL;DR: Set imagePullPolicy to Always so nodes fetch the new tag instead of reusing a cached image. The tag was reused for new content while nodes kept the old cached layers; forcing a fresh pull on every spawn resolves the mismatch. Better long term: pin images by digest or use immutable tags so a tag change can never mean two different images.

Not specified by the original author.

TL;DR

ImagePullBackOff means the kubelet cannot download the container image. The three usual causes are a typo in the image name or tag, a missing or wrong imagePullSecret for a private registry, or the tag no longer existing in the registry. Check the exact error on the pod events, then fix the image reference or the registry credentials.

Error / query

Kubernetes ImagePullBackOff causes and fixes

Use this skill when

• A pod shows ImagePullBackOff or ErrImagePull in kubectl get pods
• A new deployment never starts any containers
• kubectl describe pod shows Failed to pull image events
• A previously working deployment breaks after a tag or registry change

Not for this skill when

• The image pulls fine but the container crashes (that is CrashLoopBackOff, a different skill)
• The pod is Pending with no pull events (check scheduling, affinity, resources)
• You are pushing an image and getting registry errors (push-side auth, not pull)
• The failure is a slow pull timing out on a huge image (network/storage, not auth or naming)

Steps

Step 1: Read the exact pull error from the pod events

kubectl describe pod [pod-name] -n [namespace] | grep -A 5 "Failed"

Expected: a concrete message such as repository [name] not found, unauthorized: authentication required, or manifest unknown. The fix depends on which one you see.

Step 2: Verify the image reference actually exists in the registry

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[*].image}'

Expected: the full image string the pod is trying to pull. Compare it character by character with what is in the registry; typos in the repo name or tag are the most common cause.

Step 3: Check whether the pod has an imagePullSecret for private registries

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.imagePullSecrets}'
kubectl get secrets -n [namespace]

Expected: if the registry is private and imagePullSecrets is empty or names a secret that does not exist, that is the cause. Create a docker-registry secret and reference it.

Step 4: Create or fix the registry secret

kubectl create secret docker-registry [secret-name] -n [namespace] \
  --docker-server=[registry-host] \
  --docker-username=[username] \
  --docker-password value [registry-token]

Expected: secret created. Then add imagePullSecrets: [{name: [secret-name]}] to the pod spec or attach it to the service account, and the pull succeeds on the next attempt.

Step 5: Force a fresh pull attempt

kubectl delete pod [pod-name] -n [namespace]

Expected: the replacement pod pulls the image again. If the error persists with the same message, the image reference or credentials are still wrong, go back to steps 2-3.

Variant phrasings

"ErrImagePull kubernetes"

The transient state before the backoff kicks in. Same diagnosis; the describe output has the real error.

"unauthorized authentication required pulling image"

Registry credentials are missing, expired, or scoped to the wrong repository. Regenerate the token and update the secret.

"image not found in registry but it exists"

Usually a tag mismatch (e.g. latest vs a pinned tag) or the image living in a different repo path. Verify with the exact string from step 2.

Why it happens

The kubelet asks the registry for a manifest matching the image reference. If the name or tag does not resolve, or the registry rejects the request for missing credentials, the pull fails and the kubelet backs off retrying. The pod never starts because there is nothing to run. It is a reference or auth problem, never an application problem.

Edge cases and pitfalls

• Rate limits on public registries (e.g. Docker Hub anonymous pulls) surface as pull failures; authenticate or use a pull-through cache.
• imagePullPolicy: Always with a mutable latest tag can break a working deploy when the tag moves; pin tags for stability.
• Secrets must live in the same namespace as the pod; a secret in the wrong namespace silently does nothing.
• Private registries with self-signed certs fail pulls at the TLS layer; the error mentions certificate verification, fix the CA trust on the nodes.
• Some registries return manifest unknown for a repo the credentials cannot see, which looks like a typo; verify with a manual pull using the same credentials.

Provenance

Resolved from the public thread: https://vectle.com/posts/pster5OSbL4T0wZzAoQtsygQ

Resolved from the public thread: https://vectle.com/posts/pster5OSbL4T0wZzAoQtsygQ

Thread query: Kubernetes ImagePullBackOff causes and fixes

TL;DR: ImagePullBackOff means the kubelet cannot download the container image. The three usual causes are a typo in the image name or tag, a missing or wrong imagePullSecret for a private registry, or the tag no longer existing in the registry. Check the exact error on the pod events, then fix the image reference or the registry credentials.