## TL;DR
The cheapest way to kill rollout-time ImagePullBackOff is to have the image already on every node before the rollout starts. Run a DaemonSet (or a one-shot Job with node affinity) that pulls the target image everywhere, wait until it is Ready on all nodes, then roll out. This turns a slow registry or rate limit from a rollout blocker into a non-issue.

## Error / query
```text
how to pre-pull images on nodes to avoid ImagePullBackOff during rollouts
```

## Use this skill when
- Rollouts stall in ImagePullBackOff while images download
- The registry is slow, rate-limited, or across regions
- You need images on nodes before a maintenance window or network cutover
- Large images make every rollout take 10+ minutes in pulls

## Not for this skill when
- The pull fails with auth errors (fix credentials first; pre-pulling cannot fix auth)
- The tag does not exist (wrong tag is wrong everywhere)
- The registry itself is down (nothing to pull from)

## Steps

### Step 1: Create a pre-pull DaemonSet for the target image
```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: prepull-[name]
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: prepull-[name]
  template:
    metadata:
      labels:
        app: prepull-[name]
    spec:
      containers:
      - name: prepull
        image: [registry]/[image]:[tag]
        command: ["sh", "-c", "sleep infinity"]
        resources:
          requests:
            cpu: 10m
            memory: 16Mi
```
Expected: `kubectl apply` creates it; DaemonSet semantics put the pod on every node, which pulls the image everywhere.

### Step 2: Wait until the image is present on all nodes
```bash
kubectl rollout status daemonset/prepull-[name] -n kube-system --timeout=15m
kubectl get pods -n kube-system -l app=prepull-[name] -o wide
```
Expected: the DaemonSet reports successfully rolled out and every pod is Running. At that point the image layers exist on every node (verify spot-check with `crictl images` on a node if you want).

### Step 3: Run the real rollout, then delete the pre-pull
```bash
kubectl rollout restart deployment/[deployment] -n [namespace]
kubectl delete daemonset prepull-[name] -n kube-system
```
Expected: new pods start with the image already local, so they skip the pull entirely. Deleting the DaemonSet leaves the image cached on the nodes for future pods.

## Variant phrasings

### "warm container images on kubernetes nodes"
Same pattern. The DaemonSet in step 1 is the standard image-warming trick.

### "avoid imagepullbackoff rollout"
Pre-pull before the rollout (steps 1-2), or keep images warm continuously with a long-lived DaemonSet for hot tags.

## Why it happens
Kubelet pulls images lazily when the first pod needing them lands on a node. During a rollout, every node pulls at once, which multiplies registry load and exposes you to rate limits, slow links, and transient registry errors. Pre-pulling serializes that work ahead of time on your schedule, not during the rollout.

## Edge cases and pitfalls
- Pre-pull with the exact digest (`image@sha256:[digest]`), not just the tag, so nodes cache the exact bytes the rollout will use.
- Node disk is finite: warming many large images can trigger DiskPressure; clean stale images with an image-gc policy.
- The sleep-infinity container costs almost nothing, but delete the DaemonSet after warming or it shows up as clutter in dashboards.
- On autoscaled node groups, new nodes joining later will not have the image; re-run the pre-pull after scale-up or keep the DaemonSet around.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kE-14KjlOege9FWacSBYwg