agent stuck: hung waiting for kernel heartbeat that never arrived

TL;DR

A heartbeat that never arrives means the kernel is dead or the channel is broken, and an agent that waits indefinitely has no heartbeat policy. Set a heartbeat deadline: if no heartbeat within N seconds, mark the kernel presumed-dead, verify via the process table, then restart and resume from checkpoint. The heartbeat exists precisely to detect this; the agent must act on its absence instead of waiting through it.

agent stuck: hung waiting for kernel heartbeat that never arrived

Use this when

• An agent blocks forever waiting for a kernel heartbeat.
• The kernel process is gone but the agent never noticed.
• You own the agent's kernel monitoring.

Not for this skill when

• Heartbeats arrive but slowly. That is a loaded machine, not a hang.
• A human sees the dead kernel indicator. Restart from the UI.

Steps

1. Set a heartbeat deadline in the agent (e.g. 3 missed beats). Verify: the deadline fires on a killed test kernel.
2. On deadline, check the kernel process directly. Verify: you distinguish dead process from broken channel.
3. If dead: restart the kernel and resume from the last checkpoint. Verify: the run continues.
4. If the process lives but the channel is broken: restart the kernel session (the kernel is unreachable anyway). Verify: fresh channels work.
5. Log the incident with timestamps. Verify: post-mortems show the gap between death and detection.

Variant phrasings

agent heartbeat timeout kernel

Short form. Deadline, verify, restart.

kernel heartbeat never arrives

The symptom phrasing. Absence is the signal.

agent wait heartbeat hang

General phrasing. Every wait needs a deadline.

Compatibility: jupyter_client heartbeat channels, ipykernel 6.x. The policy is agent-side.

Why it happens

The heartbeat channel is the kernel's liveness proof, but many agent clients subscribe to it passively or not at all, treating execution replies as the only signal. When the kernel dies mid-execution there are no more replies, so the agent waits on a promise nobody can keep. A heartbeat deadline converts 'waiting' into 'detected' within seconds.

Edge cases / pitfalls

• A too-short deadline on a loaded machine declares live kernels dead. Tune from observed heartbeat intervals.
• Network partitions can break the channel while the kernel lives. The process check is what keeps this from misfiring.
• Restarting without a checkpoint just fails faster next time. Detection and resume are a pair.
• Log heartbeat gaps even when they recover. Patterns of gaps predict the next real death.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Y7WFsHfElGJTV6yONdHdvA