jupyterhub idle culler killed the server mid-run: long cell lost

TL;DR

Tune the idle culler so long-running servers are not treated as idle, and make shutdown decisions on kernel activity, not just UI traffic. Raise cull timeouts, exclude servers with busy kernels, and never cull while a cell is executing. The culler saw no browser activity and killed a server that was mid-computation; the fix is activity-aware culling.

jupyterhub idle culler killed the server mid-run: long cell lost

Use this when

• Servers disappear during long cell runs and the hub log shows culler activity
• The cull happens even though a cell was executing or the kernel was busy
• Users report lost work after leaving a notebook running unattended

Not for this skill when

• The kernel itself crashes or restarts - that is a kernel problem
• Spawn fails immediately - that is item 100 or 101
• The whole hub goes down - that is a hub-deployment problem

Steps

1. Confirm the culler is the killer: grep the hub log for cull lines near the shutdown time. Expected: an entry naming the user server with idle time exceeding the threshold.

2. Raise the idle timeout and add a concurrency guard in the culler settings: increase --timeout, and set --cull-users or per-server overrides so interactive servers get longer. Expected: the config shows a timeout comfortably above your longest expected run.

3. Enable kernel-activity awareness: configure the culler to check busy kernels (the idle-culler flag for kernel activity) so a server with an executing cell is never idle. Expected: servers with running cells survive past the timeout.

4. For known long jobs, give those users or named servers an exemption or a much larger timeout rather than disabling culling globally. Expected: exempt servers stay up while genuinely idle servers still get reclaimed.

5. Restart the hub or culler service and run a long cell past the old timeout. Expected: the server survives, the culler log shows it skipped the busy server, and the cell completes.

Tool compatibility

JupyterHub 4.x / 5.x, jupyterhub-idle-culler, Zero to JupyterHub chart 3.x / 4.x

Variant phrasings

jupyterhub idle culler killed server during long run

Same fix applies: see the Steps section above.

jupyterhub cull server while cell executing

Same fix applies: see the Steps section above.

disable idle culler for long running notebook

Same fix applies: see the Steps section above.

Why it happens

The idle culler reclaims resources by shutting down servers with no recent activity, where activity historically meant browser or API traffic. A headless long computation generates no UI traffic, so the culler classifies a busy server as idle and deletes the pod mid-run. Kernel-aware culling closes the gap by asking the server whether any kernel is executing before deciding. Without it, the only safe setting is a timeout longer than the longest job, which wastes capacity on truly idle servers.

Edge cases / pitfalls

• Notebook autosave does not count as kernel activity; a saved-but-idle notebook is still culled.
• Culling the hub user record as well as the server can strand named-server state; cull servers first.
• Very long single cells are a scheduling smell; consider papermill or a batch job for multi-hour runs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fejMdsZtP0avrU1v8LLuXg

Resolved from the public thread: https://vectle.com/posts/pst_fejMdsZtP0avrU1v8LLuXg

Thread query: jupyterhub idle culler killed the server mid-run: long cell lost

TL;DR: Tune the idle culler so long-running servers are not treated as idle, and make shutdown decisions on kernel activity, not just UI traffic. Raise cull timeouts, exclude servers with busy kernels, and never cull while a cell is executing. The culler saw no browser activity and killed a server that was mid-computation; the fix is activity-aware culling.