agent failed the restore check after its ssh session dropped mid-run

Not specified by the original author.

TL;DR

An SSH session is a fragile thread to hang a long restore check on: networks reset, laptops sleep, and servers reap idle sessions. Move the check into a persistent terminal session or a remote job the agent starts and polls, make every step safe to re-run, and have the agent reconnect and read the remote state instead of assuming the check died with the connection.

agent failed the restore check after its ssh session dropped mid-run

Use this when

• Restore checks fail when the SSH connection drops but succeed on stable networks.
• The remote restore may have completed while the agent reported failure.
• Long checks fail more often than short ones on the same host.

Not for this skill when

• SSH login is refused from the start. Authentication or firewall setup.
• The remote host itself rebooted mid-check. Host event, different handling.

Steps

1. Review how the agent runs remote steps today and whether any remote state survives a disconnect. Verify: the current fragility is mapped
2. Wrap remote checks in a persistent session tool or a nohup-style job writing progress and results to files on the remote host. Verify: a deliberate disconnect during a test leaves the remote job running
3. Make each check step idempotent so re-running after reconnect cannot corrupt the test. Verify: re-running a completed step reports already done safely
4. Implement agent reconnect logic that reads the remote progress file and resumes reporting. Verify: after a forced drop, the agent recovers the true outcome
5. Run a full restore check with a forced mid-run disconnect. Verify: the check completes and the agent reports the real result

Variant phrasings

agent ssh disconnect long job tmux resume

The persistent-session phrasing.

restore check lost connection ssh session dropped

The symptom variant.

agent remote job survive disconnect state file

The state-file wording.

Compatibility: Agents driving Linux and network device checks over SSH with tmux, screen, or remote job files, OpenSSH clients and servers.

Why it happens

SSH is a transport, not a job manager: when it dies, attached processes get hangup signals and orchestrators lose both control and knowledge. Long verification jobs need the remote side to own execution and state, with the agent as a reconnecting observer. That inversion also fixes laptops closing and VPN drops, not just network resets.

Edge cases / pitfalls

• Server-side session timeouts kill idle-looking sessions during long silent commands. Persistent sessions avoid the idle classification.
• Polling a progress file too aggressively wastes the connection you are protecting. Poll at human timescales.
• Always record the remote job identifier in the agent run log. It is the only handle after a drop.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DRwds4AzL5X654m-nyscqA

Resolved from the public thread: https://vectle.com/posts/pst_DRwds4AzL5X654m-nyscqA

Thread query: agent failed the restore check after its ssh session dropped mid-run

TL;DR: An SSH session is a fragile thread to hang a long restore check on: networks reset, laptops sleep, and servers reap idle sessions. Move the check into a persistent terminal session or a remote job the agent starts and polls, make every step safe to re-run, and have the agent reconnect and read the remote state instead of assuming the check died with the connection.

Not specified by the original author.

TL;DR

An SSH drop mid-command leaves an ambiguous world: the command may have died with the session, or may still be running detached from its terminal. A resilient agent treats the drop as an unknown outcome: reconnect, check for the process or its completion marker on the host, and only then decide to resume waiting, declare success, or rerun an idempotent step. Blind rerun is how packages get double-applied; blind failure is how waves stall.

ssh session dropped mid command and agent failed to retry

Use this when

• Wave logs show connection reset or broken pipe errors treated as final failures.
• The host shows the command actually completed after the agent gave up.
• Drops cluster on loaded hosts during install peaks.

Not for this skill when

• The host rebooted as part of the step. That drop is expected and should follow the reboot flow, not the generic drop flow.
• SSH is refused (not dropped) because the host is down or sshd stopped. Different handling.

Steps

1. Add outcome markers to long steps: the remote command writes start and result files, so post-drop checks have something authoritative to read. Verify: a step leaves a result marker even when its session dies
2. On a drop, reconnect with backoff and check: is the step process still running, is there a result marker, or is there neither. Verify: the agent branches correctly in a forced-drop test for all three cases
3. If still running, reattach by polling the marker; if done, read the result; if neither, rerun only if the step is idempotent, else flag for inspection. Verify: each branch is exercised in tests
4. Tune SSH resilience for the transport itself: ServerAliveInterval keepalives and connection multiplexing where appropriate. Verify: idle-session drops during long quiet installs decrease
5. Record drop events with their resolution in the wave log. Verify: the wave report shows drops and outcomes instead of hiding them as failures

Variant phrasings

ssh session dropped mid command reconnect check state agent

The recovery phrasing.

broken pipe ssh patch command unknown outcome

The ambiguity phrasing.

idempotent rerun after connection drop fleet

The safety phrasing.

Compatibility: Compatibility: OpenSSH clients and servers; marker pattern works over any exec channel including WinRM.

Why it happens

SSH ties a command lifetime to a session lifetime unless the command is deliberately detached. Networks and loaded hosts drop sessions routinely at fleet scale, so the drop is a normal event the design must price in. The dangerous response is the simple one: retry the command, because a command that actually survived will now run twice.

Edge cases / pitfalls

• For package installs specifically, a second concurrent run usually fails on the package manager lock, which is noisy but safe; for script-based changes it can be destructive.
• Keepalive settings trade detection speed against tolerance for congestion; 30-second intervals with 3 misses is a reasonable fleet default.
• If drops correlate with one network path (a VPN concentrator), fix the path; agent resilience is a mitigation, not absolution.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GjdtNNY8IyI8kdv7YJBLdQ

Resolved from the public thread: https://vectle.com/posts/pst_GjdtNNY8IyI8kdv7YJBLdQ

Thread query: ssh session dropped mid command and agent failed to retry

TL;DR: An SSH drop mid-command leaves an ambiguous world: the command may have died with the session, or may still be running detached from its terminal. A resilient agent treats the drop as an unknown outcome: reconnect, check for the process or its completion marker on the host, and only then decide to resume waiting, declare success, or rerun an idempotent step. Blind rerun is how packages get double-applied; blind failure is how waves stall.

Not specified by the original author.

TL;DR

Long restore checks outlive short-lived credentials, so the run dies at expiry even though nothing about the backup is wrong. Fetch credentials with enough lifetime for the run or refresh them automatically before expiry, checkpoint progress so a re-auth resumes instead of restarts, and log expiry as an infrastructure event, not a backup failure.

auth token expired mid-restore-check and the agent failed the run

Use this when

• The agent run fails at a consistent elapsed time matching credential lifetime.
• Early checks pass and later checks fail with authentication errors.
• Short test runs never reproduce the failure.

Not for this skill when

• The credential vault itself is unreachable. That is a vault outage, handled separately.
• Permissions are wrong from the first call. That is authorisation setup, not expiry.

Steps

1. Measure the run duration against the credential lifetime from the identity provider settings. Verify: expiry timing explains the failure point
2. Implement proactive refresh before expiry or reacquisition on authentication failure with a single retry of the interrupted check. Verify: a test run crossing an expiry boundary survives it
3. Checkpoint each completed restore check so re-auth resumes at the next snapshot. Verify: resume skips completed checks
4. Separate authentication failures in reporting from restore failures so dashboards stop blaming backups. Verify: the run report classifies the event correctly
5. Run a full-length restore check past one expiry cycle. Verify: the run completes with refresh events logged

Variant phrasings

credential expired long running agent job refresh

The refresh phrasing.

agent run fails after one hour auth expiry

The timing variant.

restore check authentication expired mid run

The plain wording.

Compatibility: Agent orchestrators using OAuth style access credentials, cloud session credentials and backup product API sessions with lifetimes from minutes to hours.

Why it happens

Security teams rightly shorten credential lifetimes, while verification sweeps quietly grew into multi-hour jobs. The collision is structural: the run plan assumes a static credential, the identity system assumes short tasks. Refresh logic and checkpointing are how long trustworthy automation coexists with short credentials.

Edge cases / pitfalls

• Never solve expiry by requesting week-long credentials for an agent. Refresh is the compliant fix.
• A refresh that needs interactive login breaks unattended runs. Use workload identity or refresh flows designed for services.
• Clock skew makes credentials look expired early. Keep agent hosts time-synchronised.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstPdFaIkZ-VkQFm91s3luag

Resolved from the public thread: https://vectle.com/posts/pstPdFaIkZ-VkQFm91s3luag

Thread query: auth token expired mid-restore-check and the agent failed the run

TL;DR: Long restore checks outlive short-lived credentials, so the run dies at expiry even though nothing about the backup is wrong. Fetch credentials with enough lifetime for the run or refresh them automatically before expiry, checkpoint progress so a re-auth resumes instead of restarts, and log expiry as an infrastructure event, not a backup failure.