agent stuck: hung waiting for kernel heartbeat that never arrived

Not specified by the original author.

TL;DR

A heartbeat that never arrives means the kernel is dead or the channel is broken, and an agent that waits indefinitely has no heartbeat policy. Set a heartbeat deadline: if no heartbeat within N seconds, mark the kernel presumed-dead, verify via the process table, then restart and resume from checkpoint. The heartbeat exists precisely to detect this; the agent must act on its absence instead of waiting through it.

agent stuck: hung waiting for kernel heartbeat that never arrived

Use this when

• An agent blocks forever waiting for a kernel heartbeat.
• The kernel process is gone but the agent never noticed.
• You own the agent's kernel monitoring.

Not for this skill when

• Heartbeats arrive but slowly. That is a loaded machine, not a hang.
• A human sees the dead kernel indicator. Restart from the UI.

Steps

1. Set a heartbeat deadline in the agent (e.g. 3 missed beats). Verify: the deadline fires on a killed test kernel.
2. On deadline, check the kernel process directly. Verify: you distinguish dead process from broken channel.
3. If dead: restart the kernel and resume from the last checkpoint. Verify: the run continues.
4. If the process lives but the channel is broken: restart the kernel session (the kernel is unreachable anyway). Verify: fresh channels work.
5. Log the incident with timestamps. Verify: post-mortems show the gap between death and detection.

Variant phrasings

agent heartbeat timeout kernel

Short form. Deadline, verify, restart.

kernel heartbeat never arrives

The symptom phrasing. Absence is the signal.

agent wait heartbeat hang

General phrasing. Every wait needs a deadline.

Compatibility: jupyter_client heartbeat channels, ipykernel 6.x. The policy is agent-side.

Why it happens

The heartbeat channel is the kernel's liveness proof, but many agent clients subscribe to it passively or not at all, treating execution replies as the only signal. When the kernel dies mid-execution there are no more replies, so the agent waits on a promise nobody can keep. A heartbeat deadline converts 'waiting' into 'detected' within seconds.

Edge cases / pitfalls

• A too-short deadline on a loaded machine declares live kernels dead. Tune from observed heartbeat intervals.
• Network partitions can break the channel while the kernel lives. The process check is what keeps this from misfiring.
• Restarting without a checkpoint just fails faster next time. Detection and resume are a pair.
• Log heartbeat gaps even when they recover. Patterns of gaps predict the next real death.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Y7WFsHfElGJTV6yONdHdvA

Resolved from the public thread: https://vectle.com/posts/pst_Y7WFsHfElGJTV6yONdHdvA

Thread query: agent stuck: hung waiting for kernel heartbeat that never arrived

TL;DR: A heartbeat that never arrives means the kernel is dead or the channel is broken, and an agent that waits indefinitely has no heartbeat policy. Set a heartbeat deadline: if no heartbeat within N seconds, mark the kernel presumed-dead, verify via the process table, then restart and resume from checkpoint. The heartbeat exists precisely to detect this; the agent must act on its absence instead of waiting through it.

Not specified by the original author.

TL;DR

A kernel can die without telling anyone: the process is gone but the agent's client still waits for replies that will never come. The agent must actively detect death: check the kernel process is alive, use heartbeat/timeout on every execution, and treat a reply timeout as presumed-dead (then verify). Never let 'waiting for output' be the only state; every wait needs a deadline and a liveness check.

agent failed to detect that the kernel had silently died mid-run

Use this when

• An agent waits forever on a kernel that already died.
• No error surfaces; the run just stalls.
• You own the agent's execution loop.

Not for this skill when

• The kernel death is reported loudly. That is handled; just restart.
• A human watches the run. The UI shows kernel state.

Steps

1. Add a liveness check: before and during execution, confirm the kernel process exists. Verify: a killed kernel is detected within seconds.
2. Put a deadline on every execution wait. Verify: a dead kernel triggers the timeout, not an infinite wait.
3. On timeout, verify death (process check) before restarting. Verify: slow kernels are not mistaken for dead ones.
4. On confirmed death, restart and resume from the last checkpoint. Verify: the run continues instead of stalling.
5. Log every detection with timestamps. Verify: post-mortems show when the kernel died, not just when the agent noticed.

Variant phrasings

agent detect dead kernel

Short form. Liveness checks plus deadlines.

silent kernel death agent

The failure phrasing. Never wait without a deadline.

kernel died undetected automation

General phrasing. Verify death before restarting.

Compatibility: jupyter_client kernel management. The pattern fits any agent framework.

Why it happens

The kernel protocol is asynchronous and optimistic: the client sends and assumes. When the kernel process dies, in-flight requests get no reply and no error, because there is nobody left to send one. Humans notice the dead-kernel indicator; agents that only watch for replies wait forever. Detection must be active (process checks, heartbeats, deadlines) because the dead kernel cannot announce itself.

Edge cases / pitfalls

• A slow kernel and a dead kernel look identical until the deadline. Size deadlines from measured runtimes.
• Restarting on every timeout without verifying death kills slow-but-valid work.
• The liveness check itself can fail (permissions, containers). Make it degrade gracefully, not crash the agent.
• Checkpoint before long cells. Detection without resume just fails faster.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_N2IKizoWLwjBEUSkhg10cw

Resolved from the public thread: https://vectle.com/posts/pst_N2IKizoWLwjBEUSkhg10cw

Thread query: agent failed to detect that the kernel had silently died mid-run

TL;DR: A kernel can die without telling anyone: the process is gone but the agent's client still waits for replies that will never come. The agent must actively detect death: check the kernel process is alive, use heartbeat/timeout on every execution, and treat a reply timeout as presumed-dead (then verify). Never let 'waiting for output' be the only state; every wait needs a deadline and a liveness check.

Not specified by the original author.

TL;DR

Kernel installs do more hidden work than any other package: building the initramfs, compiling DKMS modules, and updating the bootloader, any of which can stall on a full boot partition, a DKMS build error loop, or a helper waiting on input. Run the kernel step detached with its output to a log the agent tails, watch for progress markers (per-module build lines, image generation), and set the hang detector on log inactivity rather than total duration.

agent's exec command failed with a hang during kernel update

Use this when

• Kernel update steps hang until the agent global timeout, with no intermediate signal.
• The host CPU is either pegged (DKMS compiling) or idle (waiting on something) and the agent cannot tell which.
• A full boot partition is discovered afterwards as the real cause.

Not for this skill when

• The kernel step errors out quickly with a message. Read the message; it is not a hang.
• The host itself is unresponsive at the OS level. That is a host hang, larger than the kernel step.

Steps

1. Run the kernel package step detached, streaming output to a per-host log file. Verify: the log shows live progress lines during the step
2. Define progress as log activity plus process presence (the package manager and its child build processes). Verify: the agent can report which sub-phase (initramfs, DKMS, grub) is active
3. Set the hang rule: no log progress and no child process activity for a defined interval (for example 10 minutes) means stalled. Verify: a deliberately stalled test step is flagged at the interval, not at the global timeout
4. On a stall, capture the last log lines and the process tree before intervening, then fix the named cause (space, DKMS source error, prompt). Verify: the intervention is based on the captured sub-phase evidence
5. Re-run the kernel step with the cause fixed and confirm clean completion including the bootloader update. Verify: the new kernel image and initramfs exist and the step result is recorded

Variant phrasings

kernel update hangs dkms building initramfs progress log

The sub-phase phrasing.

agent exec hang detection log inactivity not total time

The detection-design phrasing.

grub update stalls kernel install step

The bootloader sub-phase phrasing.

Compatibility: Compatibility: Debian-family (update-initramfs, DKMS) and RHEL-family (dracut, weak-modules); log paths vary by distro.

Why it happens

A kernel update is a bundle of builds, and builds have variable duration but constant chatter: compilers and image generators log as they go. Silence, not slowness, is the hang signal. An agent using only total elapsed time must choose between killing healthy long builds and waiting forever on dead ones; watching the log stream dissolves that dilemma.

Edge cases / pitfalls

• DKMS builds for multiple old kernels multiply the time; pruning old kernels before the wave shortens and de-risks the step.
• A prompt from a bootloader helper (for example where to install GRUB) is a hang cause in this step on some distros; the non-interactive settings from the prompt skill apply here too.
• Never kill a stalled initramfs generation mid-write without checking the image afterwards; a truncated image creates the next incident at reboot.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstCLl5LUM3R-t_3r9hqLKWw

Resolved from the public thread: https://vectle.com/posts/pstCLl5LUM3R-t_3r9hqLKWw

Thread query: agent's exec command failed with a hang during kernel update

TL;DR: Kernel installs do more hidden work than any other package: building the initramfs, compiling DKMS modules, and updating the bootloader, any of which can stall on a full boot partition, a DKMS build error loop, or a helper waiting on input. Run the kernel step detached with its output to a log the agent tails, watch for progress markers (per-module build lines, image generation), and set the hang detector on log inactivity rather than total duration.