agent failed to detect that the kernel had silently died mid-run

TL;DR

A kernel can die without telling anyone: the process is gone but the agent's client still waits for replies that will never come. The agent must actively detect death: check the kernel process is alive, use heartbeat/timeout on every execution, and treat a reply timeout as presumed-dead (then verify). Never let 'waiting for output' be the only state; every wait needs a deadline and a liveness check.

agent failed to detect that the kernel had silently died mid-run

Use this when

• An agent waits forever on a kernel that already died.
• No error surfaces; the run just stalls.
• You own the agent's execution loop.

Not for this skill when

• The kernel death is reported loudly. That is handled; just restart.
• A human watches the run. The UI shows kernel state.

Steps

1. Add a liveness check: before and during execution, confirm the kernel process exists. Verify: a killed kernel is detected within seconds.
2. Put a deadline on every execution wait. Verify: a dead kernel triggers the timeout, not an infinite wait.
3. On timeout, verify death (process check) before restarting. Verify: slow kernels are not mistaken for dead ones.
4. On confirmed death, restart and resume from the last checkpoint. Verify: the run continues instead of stalling.
5. Log every detection with timestamps. Verify: post-mortems show when the kernel died, not just when the agent noticed.

Variant phrasings

agent detect dead kernel

Short form. Liveness checks plus deadlines.

silent kernel death agent

The failure phrasing. Never wait without a deadline.

kernel died undetected automation

General phrasing. Verify death before restarting.

Compatibility: jupyter_client kernel management. The pattern fits any agent framework.

Why it happens

The kernel protocol is asynchronous and optimistic: the client sends and assumes. When the kernel process dies, in-flight requests get no reply and no error, because there is nobody left to send one. Humans notice the dead-kernel indicator; agents that only watch for replies wait forever. Detection must be active (process checks, heartbeats, deadlines) because the dead kernel cannot announce itself.

Edge cases / pitfalls

• A slow kernel and a dead kernel look identical until the deadline. Size deadlines from measured runtimes.
• Restarting on every timeout without verifying death kills slow-but-valid work.
• The liveness check itself can fail (permissions, containers). Make it degrade gracefully, not crash the agent.
• Checkpoint before long cells. Detection without resume just fails faster.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_N2IKizoWLwjBEUSkhg10cw