## TL;DR

Set imagePullPolicy to Always so nodes fetch the new tag instead of reusing a cached image.
The tag was reused for new content while nodes kept the old cached layers; forcing a fresh pull on every spawn resolves the mismatch.
Better long term: pin images by digest or use immutable tags so a tag change can never mean two different images.

```text
jupyterhub singleuser image stuck in ImagePullBackOff after a tag change
```

## Use this when

- Singleuser pod shows ImagePullBackOff or ErrImagePull right after a tag change
- The image exists in the registry and pulls fine from a laptop
- Some nodes spawn fine and others fail, matching which nodes cached the old image

## Not for this skill when

- The pod is Pending, never reaching the pull stage - that is item 100
- The registry returns 401 or 403 - that is a registry credential problem
- Spawn times out after the image pulls - that is a resource or readiness problem

## Steps

1. Inspect the failing pod: kubectl describe pod -n YOUR_NAMESPACE [pod name].
   Expected: events showing Failed to pull image with a manifest or digest mismatch, or repeated back-off on the pull.

2. Verify the tag resolves in the registry: use crane digest or docker manifest inspect against the registry path.
   Expected: the registry returns a digest that differs from what cached nodes hold.

3. Set the pull policy in the hub config: c.KubeSpawner.image_pull_policy = 'Always' (Z2JH: singleuser: image: pullPolicy: Always). Apply with helm upgrade.
   Expected: the config renders with imagePullPolicy Always on the pod spec.

4. Delete the stuck pod so the spawner recreates it: kubectl delete pod -n YOUR_NAMESPACE [pod name].
   Expected: the replacement pod pulls the image and reaches Running.

5. For permanence, pin the image by digest in singleuser.image.name or adopt immutable tags per release.
   Expected: future deploys change the digest string, so no node can serve a stale cache under a reused tag.

## Tool compatibility

JupyterHub 4.x / 5.x, KubeSpawner 6.x, Zero to JupyterHub chart 3.x / 4.x, container registry (Docker Hub, GHCR, ECR, GAR)

## Variant phrasings

### JupyterHub ImagePullBackOff after updating singleuser image tag

Same fix applies: see the Steps section above.

### KubeSpawner ErrImagePull on new image tag

Same fix applies: see the Steps section above.

### JupyterHub pulls stale image after tag update

Same fix applies: see the Steps section above.

## Why it happens

The default imagePullPolicy IfNotPresent tells kubelet to skip pulling when the tag already exists locally. Reusing a mutable tag like latest or v2 for new content means nodes with the old layers never fetch the new ones, and nodes that do fetch it disagree with nodes that do not, producing ImagePullBackOff on the unlucky ones. The tag is a name, not a version: only the digest identifies content. Pinning by digest or setting pull policy Always removes the ambiguity, at the cost of a pull on every spawn.

## Edge cases / pitfalls

- imagePullPolicy Always slows spawns on cold nodes; combine with image pre-pulling via a daemonset for busy pools.
- A private registry needs an imagePullSecret; a 401 there looks similar but is auth, not caching.
- Digest pins must be updated in the helm values on every image release or spawns freeze on the old digest.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Dz1xjYdrHqMNS8paMsoGvw