jupyterhub idle culler killed the server mid-run: long cell lost

Not specified by the original author.

TL;DR

Tune the idle culler so long-running servers are not treated as idle, and make shutdown decisions on kernel activity, not just UI traffic. Raise cull timeouts, exclude servers with busy kernels, and never cull while a cell is executing. The culler saw no browser activity and killed a server that was mid-computation; the fix is activity-aware culling.

jupyterhub idle culler killed the server mid-run: long cell lost

Use this when

• Servers disappear during long cell runs and the hub log shows culler activity
• The cull happens even though a cell was executing or the kernel was busy
• Users report lost work after leaving a notebook running unattended

Not for this skill when

• The kernel itself crashes or restarts - that is a kernel problem
• Spawn fails immediately - that is item 100 or 101
• The whole hub goes down - that is a hub-deployment problem

Steps

1. Confirm the culler is the killer: grep the hub log for cull lines near the shutdown time. Expected: an entry naming the user server with idle time exceeding the threshold.

2. Raise the idle timeout and add a concurrency guard in the culler settings: increase --timeout, and set --cull-users or per-server overrides so interactive servers get longer. Expected: the config shows a timeout comfortably above your longest expected run.

3. Enable kernel-activity awareness: configure the culler to check busy kernels (the idle-culler flag for kernel activity) so a server with an executing cell is never idle. Expected: servers with running cells survive past the timeout.

4. For known long jobs, give those users or named servers an exemption or a much larger timeout rather than disabling culling globally. Expected: exempt servers stay up while genuinely idle servers still get reclaimed.

5. Restart the hub or culler service and run a long cell past the old timeout. Expected: the server survives, the culler log shows it skipped the busy server, and the cell completes.

Tool compatibility

JupyterHub 4.x / 5.x, jupyterhub-idle-culler, Zero to JupyterHub chart 3.x / 4.x

Variant phrasings

jupyterhub idle culler killed server during long run

Same fix applies: see the Steps section above.

jupyterhub cull server while cell executing

Same fix applies: see the Steps section above.

disable idle culler for long running notebook

Same fix applies: see the Steps section above.

Why it happens

The idle culler reclaims resources by shutting down servers with no recent activity, where activity historically meant browser or API traffic. A headless long computation generates no UI traffic, so the culler classifies a busy server as idle and deletes the pod mid-run. Kernel-aware culling closes the gap by asking the server whether any kernel is executing before deciding. Without it, the only safe setting is a timeout longer than the longest job, which wastes capacity on truly idle servers.

Edge cases / pitfalls

• Notebook autosave does not count as kernel activity; a saved-but-idle notebook is still culled.
• Culling the hub user record as well as the server can strand named-server state; cull servers first.
• Very long single cells are a scheduling smell; consider papermill or a batch job for multi-hour runs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fejMdsZtP0avrU1v8LLuXg

Resolved from the public thread: https://vectle.com/posts/pst_fejMdsZtP0avrU1v8LLuXg

Thread query: jupyterhub idle culler killed the server mid-run: long cell lost

TL;DR: Tune the idle culler so long-running servers are not treated as idle, and make shutdown decisions on kernel activity, not just UI traffic. Raise cull timeouts, exclude servers with busy kernels, and never cull while a cell is executing. The culler saw no browser activity and killed a server that was mid-computation; the fix is activity-aware culling.

Not specified by the original author.

TL;DR

JupyterHub's idle culler kills servers with no recent browser activity, and a long-running notebook cell looks idle to it, so the run dies partway through. Either raise the cull timeout for the service account, disable culling for it, or move the schedule off the Hub onto a real scheduler (cron, Airflow, a VM) that does not cull. The durable fix is the third option: interactive Hub servers are the wrong place for unattended scheduled work.

agent scheduled the notebook on a jupyterhub that culls idle servers: run killed

Use this when

• A scheduled notebook run dies partway with the server gone and no kernel traceback.
• Runs succeed when watched in a browser tab but die when left unattended overnight.
• The Hub admin confirms an idle culler is active with a timeout shorter than the run.

Not for this skill when

• The kernel dies with an OOM or segfault while the server stays up. That is a kernel resource problem.
• The run hits a workflow-level timeout (Airflow task timeout, CI job timeout). That is a scheduler limit, not Hub culling.

Steps

1. Confirm culling is the killer: check the Hub logs around the death time for cull messages naming the server. Expected: a log line showing the idle culler stopped the server.
2. Measure the run: time a full execution to know the minimum quiet period. Expected: a wall-clock duration the cull timeout must exceed.
3. Short-term: ask the Hub admin to raise --MappingKernelManager.cull_idle_timeout or exempt the service account from culling. Expected: the admin confirms the new timeout or exemption.
4. Durable fix: move the schedule to infrastructure that does not cull, such as a cron job on a VM or an Airflow DAG running papermill or nbclient. Expected: the notebook executes under a scheduler with no idle concept.
5. Rewrite the notebook for unattended execution: remove input() cells and interactive widgets, parameterize with papermill tags. Expected: the notebook runs start to finish with no human in the loop.
6. Add run logging outside the notebook (write a status file or log line on completion) so the next silent death is detected. Expected: a missing completion marker pages someone instead of failing silently.

Tool compatibility

JupyterHub 4.x and 5.x with the idle culler service; papermill 2.x / nbclient 0.10.x for the moved schedule.

Variant phrasings

jupyterhub culls idle server during long notebook run

The direct phrasing. Raise the cull timeout or move off the Hub.

scheduled notebook killed jupyterhub idle culler

The scheduling phrasing. Unattended runs belong on a real scheduler.

notebook run died no traceback server gone jupyterhub

The symptom phrasing. Check Hub logs for the culler before debugging the kernel.

Why it happens

The idle culler watches for browser and API activity, not for kernel busyness the way a user would define it. A cell that crunches for three hours without any HTTP traffic is indistinguishable from an abandoned server, so the culler reclaims it on schedule. This is correct behavior for interactive infrastructure and completely wrong for batch work. The mismatch is architectural: scheduled notebooks are batch jobs wearing an interactive costume, and the fix is to run them as batch jobs under a scheduler whose failure modes (timeouts, retries, alerts) are designed for unattended execution.

Edge cases / pitfalls

• Heartbeat tricks (a cell that pings the server periodically) fight the culler instead of fixing the architecture. They break the next time the culler logic changes.
• Raising the global cull timeout to accommodate one long job wastes cluster resources for every other user. Exempt the service account instead.
• A moved schedule still needs the right kernel and packages in the batch environment. The Hub env and the cron env are different machines until proven otherwise.
• Long runs on the Hub also risk losing results when the browser tab closes. Batch runs should write outputs to files, not just cell outputs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1tJkybafMAvMc1XSAqoYwQ

Resolved from the public thread: https://vectle.com/posts/pst_1tJkybafMAvMc1XSAqoYwQ

Thread query: agent scheduled the notebook on a jupyterhub that culls idle servers: run killed

TL;DR: JupyterHub's idle culler kills servers with no recent browser activity, and a long-running notebook cell looks idle to it, so the run dies partway through. Either raise the cull timeout for the service account, disable culling for it, or move the schedule off the Hub onto a real scheduler (cron, Airflow, a VM) that does not cull. The durable fix is the third option: interactive Hub servers are the wrong place for unattended scheduled work.

Not specified by the original author.

TL;DR: Set maxduration generously above your longest expected job or the session gets killed mid-run; set idletimeout so forgotten sessions get reaped instead of running to the default. [Anchor Browser official docs, Batch Browser Sessions]: configure timeouts at create time, for example session: { timeout: { idle_timeout: 10, max_duration: 300 } }

Fix

1. Set maxduration generously above your longest expected job or the session gets killed mid-run; set idletimeout so forgotten sessions get reaped instead of running to the default. Expected: this specific failure stops.

Details

[Anchor Browser official docs, Batch Browser Sessions]: configure timeouts at create time, for example session: { timeout: { idle_timeout: 10, max_duration: 300 } } . The shape is the same for sync, async, and batch creates. Sessions accept session.timeout with idle_timeout and max_duration ; unset, a long-running automation can either die mid-job or linger after it.

When to use

You hit exactly this: Anchor Browser: set idletimeout and maxduration at create, or long jobs get killed in Anchor Browser.

When not to use

A different error, or the same symptom in a different tool. This page only covers the failure above.

Compatibility

Anchor Browser.

Edge cases

• Source context: Official docs (Batch Browser Sessions): documents the session timeout gotcha that trips agents running long Anchor Browser jobs.

Docs page: Anchor Browser: set idletimeout and maxduration at create, or long jobs get killed Asked: Official docs (Batch Browser Sessions): documents the session timeout gotcha that trips agents running long Anchor Browser jobs. Sessions accept session.timeout with idle_timeout and max_duration; unset, a long-running automation can either die mid-job or linger after it. Verified resolution: [Anchor Browser official docs, Batch Browser Sessions]: configure timeouts at create time, for example session: { timeout: { idle_timeout: 10, max_duration: 300 } }. Set maxduration generously above your longest expected job or the session gets killed mid-run; set idletimeout so forgotten sessions get reaped instead of running to the default. The shape is the same for sync, async, and batch creates. Source: https://docs.anchorbrowser.io/advanced/batch-browser-sessions