agent scheduled the notebook on a jupyterhub that culls idle servers: run killed

TL;DR

JupyterHub's idle culler kills servers with no recent browser activity, and a long-running notebook cell looks idle to it, so the run dies partway through. Either raise the cull timeout for the service account, disable culling for it, or move the schedule off the Hub onto a real scheduler (cron, Airflow, a VM) that does not cull. The durable fix is the third option: interactive Hub servers are the wrong place for unattended scheduled work.

agent scheduled the notebook on a jupyterhub that culls idle servers: run killed

Use this when

• A scheduled notebook run dies partway with the server gone and no kernel traceback.
• Runs succeed when watched in a browser tab but die when left unattended overnight.
• The Hub admin confirms an idle culler is active with a timeout shorter than the run.

Not for this skill when

• The kernel dies with an OOM or segfault while the server stays up. That is a kernel resource problem.
• The run hits a workflow-level timeout (Airflow task timeout, CI job timeout). That is a scheduler limit, not Hub culling.

Steps

1. Confirm culling is the killer: check the Hub logs around the death time for cull messages naming the server. Expected: a log line showing the idle culler stopped the server.
2. Measure the run: time a full execution to know the minimum quiet period. Expected: a wall-clock duration the cull timeout must exceed.
3. Short-term: ask the Hub admin to raise --MappingKernelManager.cull_idle_timeout or exempt the service account from culling. Expected: the admin confirms the new timeout or exemption.
4. Durable fix: move the schedule to infrastructure that does not cull, such as a cron job on a VM or an Airflow DAG running papermill or nbclient. Expected: the notebook executes under a scheduler with no idle concept.
5. Rewrite the notebook for unattended execution: remove input() cells and interactive widgets, parameterize with papermill tags. Expected: the notebook runs start to finish with no human in the loop.
6. Add run logging outside the notebook (write a status file or log line on completion) so the next silent death is detected. Expected: a missing completion marker pages someone instead of failing silently.

Tool compatibility

JupyterHub 4.x and 5.x with the idle culler service; papermill 2.x / nbclient 0.10.x for the moved schedule.

Variant phrasings

jupyterhub culls idle server during long notebook run

The direct phrasing. Raise the cull timeout or move off the Hub.

scheduled notebook killed jupyterhub idle culler

The scheduling phrasing. Unattended runs belong on a real scheduler.

notebook run died no traceback server gone jupyterhub

The symptom phrasing. Check Hub logs for the culler before debugging the kernel.

Why it happens

The idle culler watches for browser and API activity, not for kernel busyness the way a user would define it. A cell that crunches for three hours without any HTTP traffic is indistinguishable from an abandoned server, so the culler reclaims it on schedule. This is correct behavior for interactive infrastructure and completely wrong for batch work. The mismatch is architectural: scheduled notebooks are batch jobs wearing an interactive costume, and the fix is to run them as batch jobs under a scheduler whose failure modes (timeouts, retries, alerts) are designed for unattended execution.

Edge cases / pitfalls

• Heartbeat tricks (a cell that pings the server periodically) fight the culler instead of fixing the architecture. They break the next time the culler logic changes.
• Raising the global cull timeout to accommodate one long job wastes cluster resources for every other user. Exempt the service account instead.
• A moved schedule still needs the right kernel and packages in the batch environment. The Hub env and the cron env are different machines until proven otherwise.
• Long runs on the Hub also risk losing results when the browser tab closes. Batch runs should write outputs to files, not just cell outputs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1tJkybafMAvMc1XSAqoYwQ