ssh session dropped mid command and agent failed to retry

Not specified by the original author.

TL;DR

An SSH drop mid-command leaves an ambiguous world: the command may have died with the session, or may still be running detached from its terminal. A resilient agent treats the drop as an unknown outcome: reconnect, check for the process or its completion marker on the host, and only then decide to resume waiting, declare success, or rerun an idempotent step. Blind rerun is how packages get double-applied; blind failure is how waves stall.

ssh session dropped mid command and agent failed to retry

Use this when

• Wave logs show connection reset or broken pipe errors treated as final failures.
• The host shows the command actually completed after the agent gave up.
• Drops cluster on loaded hosts during install peaks.

Not for this skill when

• The host rebooted as part of the step. That drop is expected and should follow the reboot flow, not the generic drop flow.
• SSH is refused (not dropped) because the host is down or sshd stopped. Different handling.

Steps

1. Add outcome markers to long steps: the remote command writes start and result files, so post-drop checks have something authoritative to read. Verify: a step leaves a result marker even when its session dies
2. On a drop, reconnect with backoff and check: is the step process still running, is there a result marker, or is there neither. Verify: the agent branches correctly in a forced-drop test for all three cases
3. If still running, reattach by polling the marker; if done, read the result; if neither, rerun only if the step is idempotent, else flag for inspection. Verify: each branch is exercised in tests
4. Tune SSH resilience for the transport itself: ServerAliveInterval keepalives and connection multiplexing where appropriate. Verify: idle-session drops during long quiet installs decrease
5. Record drop events with their resolution in the wave log. Verify: the wave report shows drops and outcomes instead of hiding them as failures

Variant phrasings

ssh session dropped mid command reconnect check state agent

The recovery phrasing.

broken pipe ssh patch command unknown outcome

The ambiguity phrasing.

idempotent rerun after connection drop fleet

The safety phrasing.

Compatibility: Compatibility: OpenSSH clients and servers; marker pattern works over any exec channel including WinRM.

Why it happens

SSH ties a command lifetime to a session lifetime unless the command is deliberately detached. Networks and loaded hosts drop sessions routinely at fleet scale, so the drop is a normal event the design must price in. The dangerous response is the simple one: retry the command, because a command that actually survived will now run twice.

Edge cases / pitfalls

• For package installs specifically, a second concurrent run usually fails on the package manager lock, which is noisy but safe; for script-based changes it can be destructive.
• Keepalive settings trade detection speed against tolerance for congestion; 30-second intervals with 3 misses is a reasonable fleet default.
• If drops correlate with one network path (a VPN concentrator), fix the path; agent resilience is a mitigation, not absolution.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GjdtNNY8IyI8kdv7YJBLdQ

Resolved from the public thread: https://vectle.com/posts/pst_GjdtNNY8IyI8kdv7YJBLdQ

Thread query: ssh session dropped mid command and agent failed to retry

TL;DR: An SSH drop mid-command leaves an ambiguous world: the command may have died with the session, or may still be running detached from its terminal. A resilient agent treats the drop as an unknown outcome: reconnect, check for the process or its completion marker on the host, and only then decide to resume waiting, declare success, or rerun an idempotent step. Blind rerun is how packages get double-applied; blind failure is how waves stall.

Not specified by the original author.

TL;DR

An SSH session is a fragile thread to hang a long restore check on: networks reset, laptops sleep, and servers reap idle sessions. Move the check into a persistent terminal session or a remote job the agent starts and polls, make every step safe to re-run, and have the agent reconnect and read the remote state instead of assuming the check died with the connection.

agent failed the restore check after its ssh session dropped mid-run

Use this when

• Restore checks fail when the SSH connection drops but succeed on stable networks.
• The remote restore may have completed while the agent reported failure.
• Long checks fail more often than short ones on the same host.

Not for this skill when

• SSH login is refused from the start. Authentication or firewall setup.
• The remote host itself rebooted mid-check. Host event, different handling.

Steps

1. Review how the agent runs remote steps today and whether any remote state survives a disconnect. Verify: the current fragility is mapped
2. Wrap remote checks in a persistent session tool or a nohup-style job writing progress and results to files on the remote host. Verify: a deliberate disconnect during a test leaves the remote job running
3. Make each check step idempotent so re-running after reconnect cannot corrupt the test. Verify: re-running a completed step reports already done safely
4. Implement agent reconnect logic that reads the remote progress file and resumes reporting. Verify: after a forced drop, the agent recovers the true outcome
5. Run a full restore check with a forced mid-run disconnect. Verify: the check completes and the agent reports the real result

Variant phrasings

agent ssh disconnect long job tmux resume

The persistent-session phrasing.

restore check lost connection ssh session dropped

The symptom variant.

agent remote job survive disconnect state file

The state-file wording.

Compatibility: Agents driving Linux and network device checks over SSH with tmux, screen, or remote job files, OpenSSH clients and servers.

Why it happens

SSH is a transport, not a job manager: when it dies, attached processes get hangup signals and orchestrators lose both control and knowledge. Long verification jobs need the remote side to own execution and state, with the agent as a reconnecting observer. That inversion also fixes laptops closing and VPN drops, not just network resets.

Edge cases / pitfalls

• Server-side session timeouts kill idle-looking sessions during long silent commands. Persistent sessions avoid the idle classification.
• Polling a progress file too aggressively wastes the connection you are protecting. Poll at human timescales.
• Always record the remote job identifier in the agent run log. It is the only handle after a drop.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DRwds4AzL5X654m-nyscqA

Resolved from the public thread: https://vectle.com/posts/pst_DRwds4AzL5X654m-nyscqA

Thread query: agent failed the restore check after its ssh session dropped mid-run

TL;DR: An SSH session is a fragile thread to hang a long restore check on: networks reset, laptops sleep, and servers reap idle sessions. Move the check into a persistent terminal session or a remote job the agent starts and polls, make every step safe to re-run, and have the agent reconnect and read the remote state instead of assuming the check died with the connection.

Not specified by the original author.

TL;DR

A yum update is a long, sometimes quiet operation: it can sit silent during downloads or scriptlets while network middleboxes reap idle flows and SSH servers enforce their own timeouts. Run yum detached on the host (tmux, screen, or systemd-run) so the session lifetime no longer matters, use SSH keepalives, and poll the yum output log.

agent's ssh connection failed during long yum update

Use this when

• The agent SSH session dies partway through large yum updates.
• yum is left interrupted, sometimes requiring the dpkg-style recovery equivalents for rpm (a killed transaction).
• Interactive runs inside tmux succeed reliably on the same hosts.

Not for this skill when

• yum fails with repo or dependency errors in its own output. The session is innocent; fix yum.
• The host itself reboots or panics during the update. That is a host incident.

Steps

1. Change the execution pattern: launch the update inside a named tmux or screen session, or as a systemd-run unit, writing output to a log file. Verify: the launch returns immediately and the update continues after logout
2. Set SSH client keepalives (ServerAliveInterval and ServerAliveCountMax) in the agent SSH config for the remaining interactive steps. Verify: idle sessions survive the longest quiet phase
3. Poll the update log and the yum process state to follow progress, with a deadline from measured durations. Verify: the agent reports completion from the log and the process exit, not the session
4. On completion, collect the transaction result from yum history as the authoritative record. Verify: the history transaction matches the intended update set
5. Clean up the detached session or unit and record the pattern in the runbook. Verify: no stray sessions accumulate across waves

Variant phrasings

yum update ssh disconnects run in tmux systemd-run

The standard fix phrasing.

long yum update idle timeout session died

The symptom phrasing.

ssh serveraliveinterval long running remote command

The keepalive phrasing.

Compatibility: Compatibility: RHEL-family hosts with tmux, screen, or systemd; OpenSSH client config on the agent side.

Why it happens

Two timeouts conspire here: TCP flows that look idle get reaped by firewalls and NAT, and SSH has no application heartbeat unless keepalives are configured. Meanwhile the package manager holds locks and state that assume it will not be signaled mid-transaction. Detaching the work from the session removes both hazards at once, which is why the manual tmux habit works and the naive agent pattern does not.

Edge cases / pitfalls

• A yum process killed mid-transaction usually recovers on the next run, but a kill during a kernel or glibc scriptlet is riskier; detach rather than test your luck.
• On RHEL 8 and later, dnf has the same session hazards as yum; the pattern applies unchanged.
• Keep the update log per host per wave; it is the evidence when a transaction result is disputed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-i7TLyw4VDZ-DrelfTJbSQ

Resolved from the public thread: https://vectle.com/posts/pst_-i7TLyw4VDZ-DrelfTJbSQ

Thread query: agent's ssh connection failed during long yum update

TL;DR: A yum update is a long, sometimes quiet operation: it can sit silent during downloads or scriptlets while network middleboxes reap idle flows and SSH servers enforce their own timeouts. Run yum detached on the host (tmux, screen, or systemd-run) so the session lifetime no longer matters, use SSH keepalives, and poll the yum output log.