ssh session dropped mid command and agent failed to retry

TL;DR

An SSH drop mid-command leaves an ambiguous world: the command may have died with the session, or may still be running detached from its terminal. A resilient agent treats the drop as an unknown outcome: reconnect, check for the process or its completion marker on the host, and only then decide to resume waiting, declare success, or rerun an idempotent step. Blind rerun is how packages get double-applied; blind failure is how waves stall.

ssh session dropped mid command and agent failed to retry

Use this when

• Wave logs show connection reset or broken pipe errors treated as final failures.
• The host shows the command actually completed after the agent gave up.
• Drops cluster on loaded hosts during install peaks.

Not for this skill when

• The host rebooted as part of the step. That drop is expected and should follow the reboot flow, not the generic drop flow.
• SSH is refused (not dropped) because the host is down or sshd stopped. Different handling.

Steps

1. Add outcome markers to long steps: the remote command writes start and result files, so post-drop checks have something authoritative to read. Verify: a step leaves a result marker even when its session dies
2. On a drop, reconnect with backoff and check: is the step process still running, is there a result marker, or is there neither. Verify: the agent branches correctly in a forced-drop test for all three cases
3. If still running, reattach by polling the marker; if done, read the result; if neither, rerun only if the step is idempotent, else flag for inspection. Verify: each branch is exercised in tests
4. Tune SSH resilience for the transport itself: ServerAliveInterval keepalives and connection multiplexing where appropriate. Verify: idle-session drops during long quiet installs decrease
5. Record drop events with their resolution in the wave log. Verify: the wave report shows drops and outcomes instead of hiding them as failures

Variant phrasings

ssh session dropped mid command reconnect check state agent

The recovery phrasing.

broken pipe ssh patch command unknown outcome

The ambiguity phrasing.

idempotent rerun after connection drop fleet

The safety phrasing.

Compatibility: Compatibility: OpenSSH clients and servers; marker pattern works over any exec channel including WinRM.

Why it happens

SSH ties a command lifetime to a session lifetime unless the command is deliberately detached. Networks and loaded hosts drop sessions routinely at fleet scale, so the drop is a normal event the design must price in. The dangerous response is the simple one: retry the command, because a command that actually survived will now run twice.

Edge cases / pitfalls

• For package installs specifically, a second concurrent run usually fails on the package manager lock, which is noisy but safe; for script-based changes it can be destructive.
• Keepalive settings trade detection speed against tolerance for congestion; 30-second intervals with 3 misses is a reasonable fleet default.
• If drops correlate with one network path (a VPN concentrator), fix the path; agent resilience is a mitigation, not absolution.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GjdtNNY8IyI8kdv7YJBLdQ