jupyter kernel keeps dying randomly every few minutes

Not specified by the original author.

TL;DR

Random periodic kernel deaths are the system killing the process, not your code raising: the OOM killer, a container memory limit, or an idle-culler with an aggressive timeout. Plot the kernel's memory over time and note the exact death times. If deaths coincide with memory peaks, it is OOM; if they coincide with idle periods, it is a culler; if neither, check dmesg for hardware or segfault patterns.

jupyter kernel keeps dying randomly every few minutes, how to fix

Use this when

• The kernel dies at irregular intervals with no cell running.
• Deaths happen even in an idle notebook.
• The pattern started after moving to a container, VM, or shared host.

Not for this skill when

• Deaths happen on one specific cell. That cell is the cause.
• The kernel dies exactly on a schedule. That is a culler or cron, not random.

Steps

1. Record exact death times over an hour. Verify: you have timestamps, not vibes.
2. Check for the OOM killer: dmesg | grep -i "oom\|killed process" (or the container runtime log). Verify: whether the kernel PID was OOM-killed.
3. Check for an idle culler: look at server config for cull settings and compare death times against idle periods. Verify: deaths follow idleness, not load.
4. Watch memory live: run a cell that prints psutil RSS every 30 seconds into a file. Verify: you see the growth curve before each death.
5. Fix the matching cause: raise the memory limit, disable or lengthen the culler, or fix the leaking code the graph reveals. Verify: an hour passes with no death.

Variant phrasings

jupyter kernel dies randomly

Short form. OOM vs culler vs hardware.

kernel keeps dying idle notebook

The idle variant. Cull settings are the suspect.

intermittent kernel death jupyter

General phrasing. Correlate times with logs.

Compatibility: ipykernel 6.x on Linux containers/VMs. dmesg access needs host or privileged container visibility.

Why it happens

Kernels rarely die truly randomly; the randomness is in when a threshold is crossed. Memory leaks grow until the OOM killer fires at whatever moment the limit is hit, and idle cullers fire on wall-clock timers that feel random to the user. Without timestamps and a memory curve, every theory looks equally plausible, which is why this symptom generates so much guessing.

Edge cases / pitfalls

• Restarting the kernel resets the memory curve and destroys the evidence. Measure first, restart second.
• The OOM killer may kill a sibling process (like the server) instead of the kernel. Check which PID died.
• Swap being disabled makes OOM kills sudden with no slowdown warning. The graph will show a cliff, not a slope.
• Do not raise the memory limit as the only fix when the graph shows a leak. You are just delaying the same death.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstfF1EsKMRecLq1yKiA6vIw

Resolved from the public thread: https://vectle.com/posts/pstfF1EsKMRecLq1yKiA6vIw

Thread query: jupyter kernel keeps dying randomly every few minutes, how to fix

TL;DR: Random periodic kernel deaths are the system killing the process, not your code raising: the OOM killer, a container memory limit, or an idle-culler with an aggressive timeout. Plot the kernel's memory over time and note the exact death times. If deaths coincide with memory peaks, it is OOM; if they coincide with idle periods, it is a culler; if neither, check dmesg for hardware or segfault patterns.

Not specified by the original author.

TL;DR

A kernel that restarts on every execution is not crashing randomly; something in the execution path kills it deterministically, most often a misbehaving IPython extension, %autoreload interacting badly with a module, or a kernel provisioner configured to relaunch. Disable all extensions and autoreload, then run a bare 1+1. If it survives, re-enable one at a time to find the killer.

kernel restarts and wipes variables on every cell execution, bug

Use this when

• Every cell execution triggers a restart and clears all variables.
• Even 1+1 in a fresh notebook causes a restart.
• The behavior started after installing an extension or changing config.

Not for this skill when

• Only one cell triggers a restart. That cell's code is the killer.
• The kernel restarts occasionally. That is a different, intermittent crash.

Steps

1. Disable IPython extensions: start the kernel with --InteractiveShellApp.extensions=[] or comment them out of config. Verify: 1+1 executes without restart.
2. Turn off autoreload: remove %autoreload lines and the extension. Verify: the restart stops, implicating autoreload.
3. Check the kernelspec provisioner: read kernel.json for a custom provisioner. Verify: it is the default local provisioner, not a relaunching wrapper.
4. Re-enable extensions one at a time. Verify: you identify the one whose enablement brings the restart back.
5. Remove or update the guilty extension. Verify: a full notebook run keeps variables across cells.

Variant phrasings

kernel restarts every cell execution

Short form. Extension or provisioner.

jupyter variables wiped each run

The symptom phrasing. Something relaunches per execution.

kernel auto restart on execute

General phrasing. Bisect extensions.

Compatibility: ipykernel 6.x, IPython 8/9. Extension loading is the same across versions.

Why it happens

IPython loads extensions at startup and some execute code on every cell via pre/post-run hooks. A hook that segfaults or raises SystemExit kills the kernel on each execution, and the UI restarts it, wiping variables. Autoreload is a frequent offender because it re-imports modules on every run and can execute import-time code with side effects. Custom provisioners that wrap the kernel launch can also misbehave per execution.

Edge cases / pitfalls

• The guilty extension may only crash when combined with another. Bisect in combinations if single re-enables look clean.
• Notebook metadata can auto-load extensions per notebook. Check the notebook's metadata, not just global config.
• A restart-on-every-cell can also be the OOM killer if each execution allocates hugely. Rule out memory with a trivial cell first.
• Do not reinstall everything as the first step. The bisect takes ten minutes and tells you exactly what broke.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O2bf6DsJlaZDsihcZyzEbw

Resolved from the public thread: https://vectle.com/posts/pst_O2bf6DsJlaZDsihcZyzEbw

Thread query: kernel restarts and wipes variables on every cell execution, bug

TL;DR: A kernel that restarts on every execution is not crashing randomly; something in the execution path kills it deterministically, most often a misbehaving IPython extension, %autoreload interacting badly with a module, or a kernel provisioner configured to relaunch. Disable all extensions and autoreload, then run a bare 1+1. If it survives, re-enable one at a time to find the killer.

Not specified by the original author.

TL;DR

The heartbeat timeout means the hub (or your client) stopped hearing the kernel's heartbeat replies, which in multi-user JupyterHub usually signals an overloaded node, a spawner that is slow to become ready, or network stalls, not a dead kernel. Check hub resource usage and singleuser pod events first. Then raise the heartbeat and startup timeouts to match your actual spawn times instead of fighting the defaults.

kernel heartbeat timeout error in jupyterhub multi-user setup

Use this when

• Kernels in JupyterHub report heartbeat timeouts while single-user Jupyter works.
• Timeouts cluster at busy hours or during mass spawns (class start times).
• Pod events show slow pulls or Pending states around the timeouts.

Not for this skill when

• One user's kernel always fails. That is their environment, not the hub.
• Kernels die with tracebacks. That is code or env, not heartbeat.

Steps

1. Check hub load: hub pod CPU/memory and the number of concurrent spawns. Verify: whether timeouts correlate with load spikes.
2. Check singleuser pod events: kubectl get events -n [hub-namespace]. Verify: slow pulls or scheduling delays line up with timeouts.
3. Raise the relevant timeouts in hub config (spawner start timeout, kernel heartbeat tolerance) to cover your measured spawn times. Verify: the config loads without errors.
4. Pre-pull singleuser images with a DaemonSet so spawns skip the pull. Verify: cold spawn time drops.
5. Load-test with a realistic concurrent spawn count. Verify: timeouts stay gone under peak.

Variant phrasings

jupyterhub kernel heartbeat timeout

Short form. Hub load and spawn time first.

heartbeat timed out jupyterhub

Log phrasing. Raise timeouts to measured values.

multiuser jupyter kernel timeout

General phrasing. Spawner and network are the suspects.

Compatibility: JupyterHub 3/4/5 with kubespawner or dockerspawner. Timeout knobs differ by spawner; measure first.

Why it happens

The heartbeat is a periodic are-you-alive ping between the kernel and its manager. In multi-user hubs the kernel often lives in a freshly spawned pod, so a slow image pull or a CPU-starved node delays the first heartbeat past the default timeout. The kernel is fine; it just could not answer fast enough, and the hub declares it dead.

Edge cases / pitfalls

• Raising timeouts without fixing a 10-minute image pull just makes failures slower. Pre-pull first.
• A single user's huge notebook can starve the node and cause everyone's heartbeats to lag. Set per-user resource limits.
• Network policies that block pod-to-pod traffic break heartbeats permanently. Timeouts that never clear point at policy, not load.
• Do not disable the heartbeat to silence the error. It is the mechanism that detects genuinely dead kernels.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_trKGh-NH4X3khe8G8wnR8g

Resolved from the public thread: https://vectle.com/posts/pst_trKGh-NH4X3khe8G8wnR8g

Thread query: kernel heartbeat timeout error in jupyterhub multi-user setup

TL;DR: The heartbeat timeout means the hub (or your client) stopped hearing the kernel's heartbeat replies, which in multi-user JupyterHub usually signals an overloaded node, a spawner that is slow to become ready, or network stalls, not a dead kernel. Check hub resource usage and singleuser pod events first. Then raise the heartbeat and startup timeouts to match your actual spawn times instead of fighting the defaults.