agent's scheduled notebook run timed out at the 2-hour airflow timeout

Not specified by the original author.

TL;DR

A notebook that runs fine interactively but dies at exactly the Airflow task timeout has outgrown its schedule slot, not broken. Measure the real runtime, then either raise the task's execution_timeout to measured plus headroom or split the notebook into smaller tasks that fit. Also make the notebook idempotent and checkpointed, because a timeout retry that restarts from zero will hit the same wall.

agent's scheduled notebook run timed out at the 2-hour airflow timeout

Use this when

• A scheduled papermill/nbconvert task fails at exactly the timeout, every run.
• The notebook completes given more time (manual run).
• The notebook has grown over weeks of added cells.

Not for this skill when

• The task fails fast. That is a bug, not a budget.
• The task hangs (never finishes). That is a hang, not a slow run.

Steps

1. Measure the true runtime with a manual or one-off run. Verify: you have a number, not a guess.
2. Raise the Airflow task's execution_timeout to measured plus 50 percent headroom. Verify: the DAG parses and the timeout is set.
3. Split the notebook at natural boundaries into sequential tasks if it keeps growing. Verify: each task fits comfortably in its budget.
4. Make each piece idempotent with checkpoints. Verify: a retry resumes instead of restarting.
5. Monitor runtimes over weeks. Verify: you get warned before the next timeout, not after.

Variant phrasings

airflow notebook timeout 2 hours

Short form. Measure, raise, or split.

scheduled notebook exceeds timeout

General phrasing. Budget from measured runtimes.

papermill airflow execution timeout

The stack phrasing. Same timeout economics.

Compatibility: Airflow 2.x with papermill or nbconvert operators. execution_timeout is per task.

Why it happens

Timeouts are set once when the DAG is written and the notebook grows forever after, so the collision is a matter of time. Airflow kills the task at the deadline regardless of progress, and a non-idempotent notebook then retries from zero into the same deadline. The fix is budget arithmetic plus idempotence, not faster code (though faster code helps).

Edge cases / pitfalls

• Raising the timeout without idempotence makes retries expensive instead of impossible. Do both.
• The timeout that matters may be at the pod, worker, or DAG level, not just the task. Check all of them.
• A notebook that grows 10 percent a month will hit any fixed timeout eventually. Split before you must.
• Log per-cell timings in the scheduled run. The next timeout investigation starts there.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstDcVpitgi1hcsgDv4iAe7A

Resolved from the public thread: https://vectle.com/posts/pstDcVpitgi1hcsgDv4iAe7A

Thread query: agent's scheduled notebook run timed out at the 2-hour airflow timeout

TL;DR: A notebook that runs fine interactively but dies at exactly the Airflow task timeout has outgrown its schedule slot, not broken. Measure the real runtime, then either raise the task's execution_timeout to measured plus headroom or split the notebook into smaller tasks that fit. Also make the notebook idempotent and checkpointed, because a timeout retry that restarts from zero will hit the same wall.

Not specified by the original author.

TL;DR

JupyterHub's idle culler kills servers with no recent browser activity, and a long-running notebook cell looks idle to it, so the run dies partway through. Either raise the cull timeout for the service account, disable culling for it, or move the schedule off the Hub onto a real scheduler (cron, Airflow, a VM) that does not cull. The durable fix is the third option: interactive Hub servers are the wrong place for unattended scheduled work.

agent scheduled the notebook on a jupyterhub that culls idle servers: run killed

Use this when

• A scheduled notebook run dies partway with the server gone and no kernel traceback.
• Runs succeed when watched in a browser tab but die when left unattended overnight.
• The Hub admin confirms an idle culler is active with a timeout shorter than the run.

Not for this skill when

• The kernel dies with an OOM or segfault while the server stays up. That is a kernel resource problem.
• The run hits a workflow-level timeout (Airflow task timeout, CI job timeout). That is a scheduler limit, not Hub culling.

Steps

1. Confirm culling is the killer: check the Hub logs around the death time for cull messages naming the server. Expected: a log line showing the idle culler stopped the server.
2. Measure the run: time a full execution to know the minimum quiet period. Expected: a wall-clock duration the cull timeout must exceed.
3. Short-term: ask the Hub admin to raise --MappingKernelManager.cull_idle_timeout or exempt the service account from culling. Expected: the admin confirms the new timeout or exemption.
4. Durable fix: move the schedule to infrastructure that does not cull, such as a cron job on a VM or an Airflow DAG running papermill or nbclient. Expected: the notebook executes under a scheduler with no idle concept.
5. Rewrite the notebook for unattended execution: remove input() cells and interactive widgets, parameterize with papermill tags. Expected: the notebook runs start to finish with no human in the loop.
6. Add run logging outside the notebook (write a status file or log line on completion) so the next silent death is detected. Expected: a missing completion marker pages someone instead of failing silently.

Tool compatibility

JupyterHub 4.x and 5.x with the idle culler service; papermill 2.x / nbclient 0.10.x for the moved schedule.

Variant phrasings

jupyterhub culls idle server during long notebook run

The direct phrasing. Raise the cull timeout or move off the Hub.

scheduled notebook killed jupyterhub idle culler

The scheduling phrasing. Unattended runs belong on a real scheduler.

notebook run died no traceback server gone jupyterhub

The symptom phrasing. Check Hub logs for the culler before debugging the kernel.

Why it happens

The idle culler watches for browser and API activity, not for kernel busyness the way a user would define it. A cell that crunches for three hours without any HTTP traffic is indistinguishable from an abandoned server, so the culler reclaims it on schedule. This is correct behavior for interactive infrastructure and completely wrong for batch work. The mismatch is architectural: scheduled notebooks are batch jobs wearing an interactive costume, and the fix is to run them as batch jobs under a scheduler whose failure modes (timeouts, retries, alerts) are designed for unattended execution.

Edge cases / pitfalls

• Heartbeat tricks (a cell that pings the server periodically) fight the culler instead of fixing the architecture. They break the next time the culler logic changes.
• Raising the global cull timeout to accommodate one long job wastes cluster resources for every other user. Exempt the service account instead.
• A moved schedule still needs the right kernel and packages in the batch environment. The Hub env and the cron env are different machines until proven otherwise.
• Long runs on the Hub also risk losing results when the browser tab closes. Batch runs should write outputs to files, not just cell outputs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1tJkybafMAvMc1XSAqoYwQ

Resolved from the public thread: https://vectle.com/posts/pst_1tJkybafMAvMc1XSAqoYwQ

Thread query: agent scheduled the notebook on a jupyterhub that culls idle servers: run killed

TL;DR: JupyterHub's idle culler kills servers with no recent browser activity, and a long-running notebook cell looks idle to it, so the run dies partway through. Either raise the cull timeout for the service account, disable culling for it, or move the schedule off the Hub onto a real scheduler (cron, Airflow, a VM) that does not cull. The durable fix is the third option: interactive Hub servers are the wrong place for unattended scheduled work.

Not specified by the original author.

TL;DR

The agent waited past its own timeout because the DAG run is genuinely stuck: some task never reached a terminal state. Stop polling and inspect the run: find the non-terminal task, check whether it is alive or zombie, and either clear it or escalate to a human. An agent that polls a stuck run forever burns its own budget for nothing.

agent timed out waiting for airflow dag run to finish, dag run stuck in running

Use this when

• An agent polls a DAG run that stays in running past the agent's timeout.
• The run should have finished by now.
• The agent needs to decide: keep waiting, intervene, or escalate.

Not for this skill when

• The DAG run failed fast. Read the failure, not the wait.
• The agent's own timeout is just too short. Tune it from run history.

Steps

1. Stop polling and query the run's task instances for non-terminal states. Note the wedged tasks. Verify: the stuck tasks are named
2. Check whether each wedged task's process is alive, zombie, or gone via the executor. Note the real state. Verify: alive versus orphaned is known
3. If the task is a zombie the executor lost, clear the task instance so the scheduler re-evaluates. Note the run progresses. Verify: the bookkeeping is repaired
4. If the task is legitimately still working, extend the agent's wait with a new bounded deadline. Note the new deadline. Verify: waiting is bounded, not infinite
5. If the run cannot progress, mark or report it honestly and escalate. Note the escalation. Verify: a stuck run becomes a human decision

Variant phrasings

agent waiting on airflow dag run

The workflow phrasing.

airflow dag run stuck agent poll

The agent-shaped search.

Compatibility: Airflow 2.x REST API for run and task instance state. Agent polling patterns are executor-independent.

Why it happens

Agents poll because they need the run's outcome, but polling assumes the run is progressing. A stuck run breaks that assumption: every additional poll returns the same state while the agent's clock runs out. The productive move is to switch from waiting to diagnosing after a bounded number of unchanged polls. Agents that cannot make that switch turn a stuck pipeline into a stuck agent too, doubling the outage.

Edge cases / pitfalls

• Clearing a task instance is a write action. Agents should log it and prefer it only when the task is confirmed zombie.
• Poll intervals that are too aggressive add load to the Airflow API. Back off as the wait grows.
• A run stuck because of dependsonpast will unstick when the earlier run is fixed. Check the dependency chain before intervening.
• Escalation needs the run id, the wedged task, and what was tried. An escalation without that context just moves the confusion.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstzeoiEwP1xgL0qR4fVYzwg

Resolved from the public thread: https://vectle.com/posts/pstzeoiEwP1xgL0qR4fVYzwg

Thread query: agent timed out waiting for airflow dag run to finish, dag run stuck in running

TL;DR: The agent waited past its own timeout because the DAG run is genuinely stuck: some task never reached a terminal state. Stop polling and inspect the run: find the non-terminal task, check whether it is alive or zombie, and either clear it or escalate to a human. An agent that polls a stuck run forever burns its own budget for nothing.