how to pre-pull images on nodes to avoid ImagePullBackOff during rollouts

TL;DR

The cheapest way to kill rollout-time ImagePullBackOff is to have the image already on every node before the rollout starts. Run a DaemonSet (or a one-shot Job with node affinity) that pulls the target image everywhere, wait until it is Ready on all nodes, then roll out. This turns a slow registry or rate limit from a rollout blocker into a non-issue.

Error / query

how to pre-pull images on nodes to avoid ImagePullBackOff during rollouts

Use this skill when

• Rollouts stall in ImagePullBackOff while images download
• The registry is slow, rate-limited, or across regions
• You need images on nodes before a maintenance window or network cutover
• Large images make every rollout take 10+ minutes in pulls

Not for this skill when

• The pull fails with auth errors (fix credentials first; pre-pulling cannot fix auth)
• The tag does not exist (wrong tag is wrong everywhere)
• The registry itself is down (nothing to pull from)

Steps

Step 1: Create a pre-pull DaemonSet for the target image

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: prepull-[name]
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: prepull-[name]
  template:
    metadata:
      labels:
        app: prepull-[name]
    spec:
      containers:
      - name: prepull
        image: [registry]/[image]:[tag]
        command: ["sh", "-c", "sleep infinity"]
        resources:
          requests:
            cpu: 10m
            memory: 16Mi

Expected: kubectl apply creates it; DaemonSet semantics put the pod on every node, which pulls the image everywhere.

Step 2: Wait until the image is present on all nodes

kubectl rollout status daemonset/prepull-[name] -n kube-system --timeout=15m
kubectl get pods -n kube-system -l app=prepull-[name] -o wide

Expected: the DaemonSet reports successfully rolled out and every pod is Running. At that point the image layers exist on every node (verify spot-check with crictl images on a node if you want).

Step 3: Run the real rollout, then delete the pre-pull

kubectl rollout restart deployment/[deployment] -n [namespace]
kubectl delete daemonset prepull-[name] -n kube-system

Expected: new pods start with the image already local, so they skip the pull entirely. Deleting the DaemonSet leaves the image cached on the nodes for future pods.

Variant phrasings

"warm container images on kubernetes nodes"

Same pattern. The DaemonSet in step 1 is the standard image-warming trick.

"avoid imagepullbackoff rollout"

Pre-pull before the rollout (steps 1-2), or keep images warm continuously with a long-lived DaemonSet for hot tags.

Why it happens

Kubelet pulls images lazily when the first pod needing them lands on a node. During a rollout, every node pulls at once, which multiplies registry load and exposes you to rate limits, slow links, and transient registry errors. Pre-pulling serializes that work ahead of time on your schedule, not during the rollout.

Edge cases and pitfalls

• Pre-pull with the exact digest (image@sha256:[digest]), not just the tag, so nodes cache the exact bytes the rollout will use.
• Node disk is finite: warming many large images can trigger DiskPressure; clean stale images with an image-gc policy.
• The sleep-infinity container costs almost nothing, but delete the DaemonSet after warming or it shows up as clutter in dashboards.
• On autoscaled node groups, new nodes joining later will not have the image; re-run the pre-pull after scale-up or keep the DaemonSet around.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kE-14KjlOege9FWacSBYwg