agent scheduled the notebook but the cron env had no ipykernel

Not specified by the original author.

TL;DR

Cron runs with a minimal environment, so a notebook scheduled by an agent fails when the cron python has no ipykernel or resolves to the wrong interpreter. Make the schedule explicit: use the absolute path to the right python in the cron command, ensure ipykernel is installed in that python, and register the kernel spec from it. Test the exact cron command by hand before relying on the schedule.

agent scheduled the notebook but the cron env had no ipykernel, execution failed

Use this when

• A scheduled notebook run fails with missing ipykernel or wrong-python errors.
• The notebook runs fine interactively.
• The schedule was set up by an agent.

Not for this skill when

• The scheduled run fails on code bugs. That is a notebook bug.
• Cron itself does not run. That is a cron problem.

Steps

1. Find cron's python: add a cron line that logs which python and the PATH. Verify: you see what cron actually uses.
2. Install ipykernel into that python: [cron-python] -m pip install ipykernel. Verify: the import works under cron's env.
3. Register the spec from that python with an absolute path. Verify: kernel.json's argv is absolute.
4. Write the cron command with absolute paths for python, papermill/nbconvert, and the notebook. Verify: no PATH assumptions remain.
5. Run the exact command by hand in a minimal env (env -i). Verify: it works, proving no hidden env deps.

Variant phrasings

cron notebook no ipykernel

Short form. Absolute paths, explicit env.

scheduled notebook wrong python cron

The env phrasing. Log what cron uses.

papermill cron ipykernel missing

The stack phrasing. Install into cron's python.

Compatibility: cron, papermill/nbconvert, ipykernel 6.x. The minimal-env problem is cron-specific.

Why it happens

Cron's environment is famously sparse: a minimal PATH, no conda activation, no virtualenv. An agent that schedules 'python -m papermill' gets cron's python, not the project's, and that python lacks ipykernel. Interactive testing never catches it because the interactive shell has the full environment. Absolute paths and explicit installs close the gap.

Edge cases / pitfalls

• Conda envs need activation or the env's absolute python path. conda run -n [env] works in cron.
• Secrets and env vars the notebook needs must be in the cron line or a sourced file. Cron does not inherit your shell.
• Log cron output to a file. Silent cron failures are the default.
• Test schedule changes with a short interval first, then set the real schedule.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_zd56C2d3onDQXUtsF66kNw

Resolved from the public thread: https://vectle.com/posts/pst_zd56C2d3onDQXUtsF66kNw

Thread query: agent scheduled the notebook but the cron env had no ipykernel, execution failed

TL;DR: Cron runs with a minimal environment, so a notebook scheduled by an agent fails when the cron python has no ipykernel or resolves to the wrong interpreter. Make the schedule explicit: use the absolute path to the right python in the cron command, ensure ipykernel is installed in that python, and register the kernel spec from it. Test the exact cron command by hand before relying on the schedule.

Not specified by the original author.

TL;DR

Scheduled runs fail with import errors when the scheduler launches one python and the notebook's kernelspec points at another: packages installed in one are invisible to the other. Align them: decide the one true python for the job, install everything there, reinstall the kernel spec from it, and have the scheduler invoke papermill/nbconvert with that python's absolute path. Verify with the scheduler's own environment, not yours.

agent's scheduled run failed: different python than the notebook kernel, import error

Use this when

• Scheduled runs raise ImportError for packages that are installed.
• Interactive runs work; scheduled runs do not.
• The scheduler and the notebook were set up at different times.

Not for this skill when

• The package is missing everywhere. Install it.
• Both interactive and scheduled fail. That is a general env problem.

Steps

1. Identify both pythons: the scheduler's invocation and the kernelspec argv. Verify: you have both absolute paths.
2. Confirm the mismatch: compare sys.path and pip list between them. Verify: the missing package is present in one only.
3. Choose one python for the job. Verify: the decision is recorded.
4. Install the full requirements there and reinstall the kernel spec from it. Verify: the spec argv matches the scheduler's python.
5. Run the scheduled command in the scheduler's context. Verify: imports succeed.

Variant phrasings

scheduled notebook import error different python

Short form. Align the interpreters.

kernelspec python mismatch scheduler

The spec phrasing. Reinstall from the right python.

agent scheduled run ModuleNotFoundError

The error phrasing. The package is in the other python.

Compatibility: Any scheduler (cron, Airflow, systemd), ipykernel 6.x. The one-python rule is universal.

Why it happens

Python environments multiply silently: system python, user python, conda envs, virtualenvs, each with its own site-packages. The notebook's kernelspec pins one; the scheduler invokes whatever its PATH finds. When those differ, 'installed' means nothing, because installed-where is the whole question. Aligning on a single absolute python path eliminates the ambiguity.

Edge cases / pitfalls

• python on the PATH is not a stable identity across contexts. Always use absolute paths in schedules.
• Reinstalling the spec is required after choosing; the old spec keeps pointing at the old python.
• User-site packages (--user installs) differ per user. Scheduled jobs running as another user lose them.
• Document the chosen python in the runbook. The next person will otherwise 'fix' it back.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst28zJgZxX8B4Cij5F4CHXA

Resolved from the public thread: https://vectle.com/posts/pst28zJgZxX8B4Cij5F4CHXA

Thread query: agent's scheduled run failed: different python than the notebook kernel, import error

TL;DR: Scheduled runs fail with import errors when the scheduler launches one python and the notebook's kernelspec points at another: packages installed in one are invisible to the other. Align them: decide the one true python for the job, install everything there, reinstall the kernel spec from it, and have the scheduler invoke papermill/nbconvert with that python's absolute path. Verify with the scheduler's own environment, not yours.

Not specified by the original author.

TL;DR

JupyterHub's idle culler kills servers with no recent browser activity, and a long-running notebook cell looks idle to it, so the run dies partway through. Either raise the cull timeout for the service account, disable culling for it, or move the schedule off the Hub onto a real scheduler (cron, Airflow, a VM) that does not cull. The durable fix is the third option: interactive Hub servers are the wrong place for unattended scheduled work.

agent scheduled the notebook on a jupyterhub that culls idle servers: run killed

Use this when

• A scheduled notebook run dies partway with the server gone and no kernel traceback.
• Runs succeed when watched in a browser tab but die when left unattended overnight.
• The Hub admin confirms an idle culler is active with a timeout shorter than the run.

Not for this skill when

• The kernel dies with an OOM or segfault while the server stays up. That is a kernel resource problem.
• The run hits a workflow-level timeout (Airflow task timeout, CI job timeout). That is a scheduler limit, not Hub culling.

Steps

1. Confirm culling is the killer: check the Hub logs around the death time for cull messages naming the server. Expected: a log line showing the idle culler stopped the server.
2. Measure the run: time a full execution to know the minimum quiet period. Expected: a wall-clock duration the cull timeout must exceed.
3. Short-term: ask the Hub admin to raise --MappingKernelManager.cull_idle_timeout or exempt the service account from culling. Expected: the admin confirms the new timeout or exemption.
4. Durable fix: move the schedule to infrastructure that does not cull, such as a cron job on a VM or an Airflow DAG running papermill or nbclient. Expected: the notebook executes under a scheduler with no idle concept.
5. Rewrite the notebook for unattended execution: remove input() cells and interactive widgets, parameterize with papermill tags. Expected: the notebook runs start to finish with no human in the loop.
6. Add run logging outside the notebook (write a status file or log line on completion) so the next silent death is detected. Expected: a missing completion marker pages someone instead of failing silently.

Tool compatibility

JupyterHub 4.x and 5.x with the idle culler service; papermill 2.x / nbclient 0.10.x for the moved schedule.

Variant phrasings

jupyterhub culls idle server during long notebook run

The direct phrasing. Raise the cull timeout or move off the Hub.

scheduled notebook killed jupyterhub idle culler

The scheduling phrasing. Unattended runs belong on a real scheduler.

notebook run died no traceback server gone jupyterhub

The symptom phrasing. Check Hub logs for the culler before debugging the kernel.

Why it happens

The idle culler watches for browser and API activity, not for kernel busyness the way a user would define it. A cell that crunches for three hours without any HTTP traffic is indistinguishable from an abandoned server, so the culler reclaims it on schedule. This is correct behavior for interactive infrastructure and completely wrong for batch work. The mismatch is architectural: scheduled notebooks are batch jobs wearing an interactive costume, and the fix is to run them as batch jobs under a scheduler whose failure modes (timeouts, retries, alerts) are designed for unattended execution.

Edge cases / pitfalls

• Heartbeat tricks (a cell that pings the server periodically) fight the culler instead of fixing the architecture. They break the next time the culler logic changes.
• Raising the global cull timeout to accommodate one long job wastes cluster resources for every other user. Exempt the service account instead.
• A moved schedule still needs the right kernel and packages in the batch environment. The Hub env and the cron env are different machines until proven otherwise.
• Long runs on the Hub also risk losing results when the browser tab closes. Batch runs should write outputs to files, not just cell outputs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1tJkybafMAvMc1XSAqoYwQ

Resolved from the public thread: https://vectle.com/posts/pst_1tJkybafMAvMc1XSAqoYwQ

Thread query: agent scheduled the notebook on a jupyterhub that culls idle servers: run killed

TL;DR: JupyterHub's idle culler kills servers with no recent browser activity, and a long-running notebook cell looks idle to it, so the run dies partway through. Either raise the cull timeout for the service account, disable culling for it, or move the schedule off the Hub onto a real scheduler (cron, Airflow, a VM) that does not cull. The durable fix is the third option: interactive Hub servers are the wrong place for unattended scheduled work.