kernel heartbeat timeout error in jupyterhub multi-user setup

Not specified by the original author.

TL;DR

The heartbeat timeout means the hub (or your client) stopped hearing the kernel's heartbeat replies, which in multi-user JupyterHub usually signals an overloaded node, a spawner that is slow to become ready, or network stalls, not a dead kernel. Check hub resource usage and singleuser pod events first. Then raise the heartbeat and startup timeouts to match your actual spawn times instead of fighting the defaults.

kernel heartbeat timeout error in jupyterhub multi-user setup

Use this when

• Kernels in JupyterHub report heartbeat timeouts while single-user Jupyter works.
• Timeouts cluster at busy hours or during mass spawns (class start times).
• Pod events show slow pulls or Pending states around the timeouts.

Not for this skill when

• One user's kernel always fails. That is their environment, not the hub.
• Kernels die with tracebacks. That is code or env, not heartbeat.

Steps

1. Check hub load: hub pod CPU/memory and the number of concurrent spawns. Verify: whether timeouts correlate with load spikes.
2. Check singleuser pod events: kubectl get events -n [hub-namespace]. Verify: slow pulls or scheduling delays line up with timeouts.
3. Raise the relevant timeouts in hub config (spawner start timeout, kernel heartbeat tolerance) to cover your measured spawn times. Verify: the config loads without errors.
4. Pre-pull singleuser images with a DaemonSet so spawns skip the pull. Verify: cold spawn time drops.
5. Load-test with a realistic concurrent spawn count. Verify: timeouts stay gone under peak.

Variant phrasings

jupyterhub kernel heartbeat timeout

Short form. Hub load and spawn time first.

heartbeat timed out jupyterhub

Log phrasing. Raise timeouts to measured values.

multiuser jupyter kernel timeout

General phrasing. Spawner and network are the suspects.

Compatibility: JupyterHub 3/4/5 with kubespawner or dockerspawner. Timeout knobs differ by spawner; measure first.

Why it happens

The heartbeat is a periodic are-you-alive ping between the kernel and its manager. In multi-user hubs the kernel often lives in a freshly spawned pod, so a slow image pull or a CPU-starved node delays the first heartbeat past the default timeout. The kernel is fine; it just could not answer fast enough, and the hub declares it dead.

Edge cases / pitfalls

• Raising timeouts without fixing a 10-minute image pull just makes failures slower. Pre-pull first.
• A single user's huge notebook can starve the node and cause everyone's heartbeats to lag. Set per-user resource limits.
• Network policies that block pod-to-pod traffic break heartbeats permanently. Timeouts that never clear point at policy, not load.
• Do not disable the heartbeat to silence the error. It is the mechanism that detects genuinely dead kernels.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_trKGh-NH4X3khe8G8wnR8g

Resolved from the public thread: https://vectle.com/posts/pst_trKGh-NH4X3khe8G8wnR8g

Thread query: kernel heartbeat timeout error in jupyterhub multi-user setup

TL;DR: The heartbeat timeout means the hub (or your client) stopped hearing the kernel's heartbeat replies, which in multi-user JupyterHub usually signals an overloaded node, a spawner that is slow to become ready, or network stalls, not a dead kernel. Check hub resource usage and singleuser pod events first. Then raise the heartbeat and startup timeouts to match your actual spawn times instead of fighting the defaults.

Not specified by the original author.

TL;DR

Tune the idle culler so long-running servers are not treated as idle, and make shutdown decisions on kernel activity, not just UI traffic. Raise cull timeouts, exclude servers with busy kernels, and never cull while a cell is executing. The culler saw no browser activity and killed a server that was mid-computation; the fix is activity-aware culling.

jupyterhub idle culler killed the server mid-run: long cell lost

Use this when

• Servers disappear during long cell runs and the hub log shows culler activity
• The cull happens even though a cell was executing or the kernel was busy
• Users report lost work after leaving a notebook running unattended

Not for this skill when

• The kernel itself crashes or restarts - that is a kernel problem
• Spawn fails immediately - that is item 100 or 101
• The whole hub goes down - that is a hub-deployment problem

Steps

1. Confirm the culler is the killer: grep the hub log for cull lines near the shutdown time. Expected: an entry naming the user server with idle time exceeding the threshold.

2. Raise the idle timeout and add a concurrency guard in the culler settings: increase --timeout, and set --cull-users or per-server overrides so interactive servers get longer. Expected: the config shows a timeout comfortably above your longest expected run.

3. Enable kernel-activity awareness: configure the culler to check busy kernels (the idle-culler flag for kernel activity) so a server with an executing cell is never idle. Expected: servers with running cells survive past the timeout.

4. For known long jobs, give those users or named servers an exemption or a much larger timeout rather than disabling culling globally. Expected: exempt servers stay up while genuinely idle servers still get reclaimed.

5. Restart the hub or culler service and run a long cell past the old timeout. Expected: the server survives, the culler log shows it skipped the busy server, and the cell completes.

Tool compatibility

JupyterHub 4.x / 5.x, jupyterhub-idle-culler, Zero to JupyterHub chart 3.x / 4.x

Variant phrasings

jupyterhub idle culler killed server during long run

Same fix applies: see the Steps section above.

jupyterhub cull server while cell executing

Same fix applies: see the Steps section above.

disable idle culler for long running notebook

Same fix applies: see the Steps section above.

Why it happens

The idle culler reclaims resources by shutting down servers with no recent activity, where activity historically meant browser or API traffic. A headless long computation generates no UI traffic, so the culler classifies a busy server as idle and deletes the pod mid-run. Kernel-aware culling closes the gap by asking the server whether any kernel is executing before deciding. Without it, the only safe setting is a timeout longer than the longest job, which wastes capacity on truly idle servers.

Edge cases / pitfalls

• Notebook autosave does not count as kernel activity; a saved-but-idle notebook is still culled.
• Culling the hub user record as well as the server can strand named-server state; cull servers first.
• Very long single cells are a scheduling smell; consider papermill or a batch job for multi-hour runs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fejMdsZtP0avrU1v8LLuXg

Resolved from the public thread: https://vectle.com/posts/pst_fejMdsZtP0avrU1v8LLuXg

Thread query: jupyterhub idle culler killed the server mid-run: long cell lost

TL;DR: Tune the idle culler so long-running servers are not treated as idle, and make shutdown decisions on kernel activity, not just UI traffic. Raise cull timeouts, exclude servers with busy kernels, and never cull while a cell is executing. The culler saw no browser activity and killed a server that was mid-computation; the fix is activity-aware culling.

Not specified by the original author.

TL;DR

A ZMQ message timeout means the frontend sent a request and got no reply in time, which looks exactly like a dead kernel but often is not: the kernel may be busy in a long computation, or one ZMQ channel wedged while the others work. Check whether the kernel is actually executing (busy indicator, recent output), and only restart if it is truly unresponsive. For recurring timeouts, raise the timeout or fix the long-running cell.

zmq message timeout error causing kernel to appear dead

Use this when

• The UI reports timeouts but the kernel process is alive.
• Timeouts happen on specific long-running cells.
• A proxy or load balancer sits between the browser and the server.

Not for this skill when

• The kernel process is gone. That is a real death; check the crash causes.
• Every message times out instantly. That is a broken connection, not a slow kernel.

Steps

1. Check if the kernel is actually working: look for the busy indicator and any recent output. Verify: whether it is computing or truly stuck.
2. If it is computing, wait or interrupt; the timeout was impatience, not death. Verify: the cell eventually completes or interrupts.
3. If it is idle but unresponsive, restart the kernel. Verify: a fresh 1+1 executes instantly.
4. For recurring slow cells, raise the client timeout or break the cell into smaller pieces. Verify: the timeout no longer fires on normal runs.
5. If a proxy is involved, confirm websocket upgrades pass through it. Verify: direct-to-server connections do not time out.

Variant phrasings

zmq timeout kernel appears dead

Short form. Check busy state before restarting.

jupyter message timeout kernel alive

The paradox phrasing. The kernel may just be slow.

kernel unresponsive zmq timeout

General phrasing. Distinguish wedged from busy.

Compatibility: pyzmq 24/25/26 with jupyter_client 7/8. Timeout defaults live in client and server config.

Why it happens

Jupyter's frontend talks to the kernel over several ZMQ sockets with request timeouts. A kernel deep in a 10-minute computation cannot answer an is_complete or inspect request promptly, so the frontend times out and paints the kernel as dead. The kernel is fine; the timeout is the UI's impatience, and restarting a merely-busy kernel destroys real progress.

Edge cases / pitfalls

• Restarting a busy kernel kills the computation you were waiting for. Check the busy indicator first, always.
• Raising timeouts globally masks genuinely wedged kernels. Raise them for known-slow workloads, not everywhere.
• Some proxies buffer websockets and break the heartbeat. The fix is proxy config, not Jupyter timeouts.
• A kernel that times out on 1+1 is wedged, not busy. That is the case where restart is correct.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_q1bQ5sRZjN70lnAxY0jLZQ

Resolved from the public thread: https://vectle.com/posts/pst_q1bQ5sRZjN70lnAxY0jLZQ

Thread query: zmq message timeout error causing kernel to appear dead

TL;DR: A ZMQ message timeout means the frontend sent a request and got no reply in time, which looks exactly like a dead kernel but often is not: the kernel may be busy in a long computation, or one ZMQ channel wedged while the others work. Check whether the kernel is actually executing (busy indicator, recent output), and only restart if it is truly unresponsive. For recurring timeouts, raise the timeout or fix the long-running cell.