jupyterhub spawn failed: timeout waiting for the pod to start on a full node

TL;DR

Raise the spawner start timeout and give the autoscaler room to add capacity so Pending pods can schedule. The immediate fix is KubeSpawner.start_timeout raised to 600 seconds plus lower resource requests or node taints so the pod finds a node before the hub gives up. Root cause is usually a full node: the pod waits for a slot that never appears inside the default timeout.

jupyterhub spawn failed: timeout waiting for the pod to start on a full node

Use this when

• Spawn fails with a timeout and the singleuser pod shows status Pending
• The node pool is at capacity and cluster-autoscaler events show scale-up pending
• Spawns succeed at off-peak hours but time out during busy periods

Not for this skill when

• The pod is stuck in ImagePullBackOff or ErrImagePull - that is item 102
• Spawn fails with 403 before a pod is created - that is item 101
• The hub itself is unreachable - that is a hub-deployment problem

Steps

1. Check why the pod is Pending with: kubectl describe pod -n YOUR_NAMESPACE [singleuser pod name]. Expected: events showing FailedScheduling, Insufficient cpu or Insufficient memory, naming the missing resource. Expected: The pod name looks like jupyter-username. Copy the full event text before changing anything.

2. Confirm node pressure with: kubectl top nodes and kubectl get nodes. Expected: most nodes show cpu or memory above 85 percent, matching the FailedScheduling event.

3. Raise the spawner start timeout in the hub config: c.KubeSpawner.start_timeout = 600. Apply with helm upgrade and restart the hub. Expected: the hub waits 10 minutes for the pod before declaring a timeout.

4. Lower or right-size the per-user requests: c.KubeSpawner.cpuguarantee and c.KubeSpawner.memguarantee set below the node headroom, or add a dedicated user node pool with a toleration. Expected: kubectl describe pod shows the pod scheduled within a few minutes.

5. If cluster-autoscaler is enabled, verify it can add nodes: kubectl get events -n kube-system filtered for autoscaler. Expected: SuccessfulTriggerScaleUp events within 1 to 2 minutes of the spawn attempt.

6. Retry the spawn from the hub control panel. Expected: the server reaches Running status and the notebook UI loads, spawn progress completing inside the new timeout.

Tool compatibility

JupyterHub 4.x / 5.x, KubeSpawner 6.x, Zero to JupyterHub chart 3.x / 4.x, Kubernetes 1.28+, cluster-autoscaler

Variant phrasings

JupyterHub spawn timed out waiting for pod to become ready

Same fix applies: see the Steps section above.

Singleuser pod stuck Pending, hub reports spawn failed

Same fix applies: see the Steps section above.

JupyterHub timeout scheduling user server on Kubernetes

Same fix applies: see the Steps section above.

Why it happens

KubeSpawner asks the Kubernetes scheduler for a pod and waits start_timeout seconds (default 300) for it to reach Running. On a full node pool the pod sits in Pending with FailedScheduling events, and the hub kills the attempt when the timer expires even though nothing is broken. This repeats at peak hours because the scheduler cannot conjure capacity. Raising the timeout only helps if the autoscaler can actually add nodes; otherwise the requests must shrink or a dedicated pool must absorb the load. The event stream on the Pending pod is the source of truth for which resource is missing.

Edge cases / pitfalls

• Raising the timeout without autoscaler headroom just makes users wait longer before the same failure.
• cpu_guarantee values above what a single node can offer make Pending permanent, not slow.
• Preemptible or spot node pools can evict the pod mid-session; keep user servers on on-demand nodes.
• Check the hub pod logs too; a 500 from the hub during spawn looks similar but is a different failure.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstSeCp5S8U9wG_-LoCetYCA