jupyterhub proxy error 503 after a hub restart: routes not re-registered

TL;DR

Restart or re-register the orphaned singleuser servers so the configurable proxy learns their routes again. The hub restart wiped the proxy route table; Running servers whose routes were lost return 503 until their routes are re-added. Fix it by stopping the orphaned servers from the admin panel (or API) and letting users respawn, which re-registers routes cleanly.

jupyterhub proxy error 503 after a hub restart: routes not re-registered

Use this when

• Proxy returns 503 for user servers right after a hub restart or upgrade
• kubectl shows singleuser pods Running but the hub UI cannot reach them
• The hub log shows no route found or proxy warns about missing targets

Not for this skill when

• The 503 appears during a fresh spawn that never reached Running - that is item 100
• Users get 403 instead of 503 - that is item 101
• The hub pod itself is crash-looping - that is a hub-deployment problem

Steps

1. Confirm the route table is empty: check the hub log for proxy route warnings, or query the proxy API at the hub service proxy path with the proxy auth. Expected: user routes missing while pods are Running.

2. As admin, open the hub admin panel and stop each orphaned server. Expected: the pods terminate and the stale route entries are cleared.

3. Ask users to start their servers again from the control panel. Expected: each spawn re-registers its route and the notebook UI loads without 503.

4. If servers must survive hub restarts, enable named-server persistence settings and consider the hub database surviving restarts (sqlite on a persistent volume or an external database). Expected: after the next hub restart, existing servers keep their routes.

5. Verify end to end: curl the user server URL through the proxy. Expected: HTTP 200 and the JupyterLab HTML, not a 503 page.

Tool compatibility

JupyterHub 4.x / 5.x, configurable-http-proxy 4.x, Zero to JupyterHub chart 3.x / 4.x

Variant phrasings

JupyterHub 503 proxy error after hub restart

Same fix applies: see the Steps section above.

configurable-http-proxy no route after hub upgrade

Same fix applies: see the Steps section above.

JupyterHub user server 503 but pod is running

Same fix applies: see the Steps section above.

Why it happens

The configurable proxy keeps its route table in memory, populated by the hub as servers spawn. Restarting the hub process clears that table, and the hub only re-adds routes for servers it actively manages through a fresh spawn. Singleuser pods that survived the restart are orphans: alive, but with no route pointing at them, so the proxy answers 503. Stopping the orphans and respawning re-registers everything. Persisting the hub database across restarts helps the hub remember the servers, but the route table itself still needs the spawn handshake.

Edge cases / pitfalls

• Deleting the hub pod without a persistent database loses all server state; respawn is then the only path.
• Named servers add route prefixes; verify each named server route individually, not just the default one.
• A 503 from the ingress in front of the proxy is a different layer; check the proxy response headers to tell them apart.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CIb9zoD7YQEQ5BQPobYoMQ

Resolved from the public thread: https://vectle.com/posts/pst_CIb9zoD7YQEQ5BQPobYoMQ

Thread query: jupyterhub proxy error 503 after a hub restart: routes not re-registered

TL;DR: Restart or re-register the orphaned singleuser servers so the configurable proxy learns their routes again. The hub restart wiped the proxy route table; Running servers whose routes were lost return 503 until their routes are re-added. Fix it by stopping the orphaned servers from the admin panel (or API) and letting users respawn, which re-registers routes cleanly.