ssh session dropped mid restore check

Not specified by the original author.

TL;DR

An SSH session is a fragile thread to hang a long restore check on: networks reset, laptops sleep, and servers reap idle sessions. Move the check into a persistent terminal session or a remote job the agent starts and polls, make every step safe to re-run, and have the agent reconnect and read the remote state instead of assuming the check died with the connection.

agent failed the restore check after its ssh session dropped mid-run

Use this when

• Restore checks fail when the SSH connection drops but succeed on stable networks.
• The remote restore may have completed while the agent reported failure.
• Long checks fail more often than short ones on the same host.

Not for this skill when

• SSH login is refused from the start. Authentication or firewall setup.
• The remote host itself rebooted mid-check. Host event, different handling.

Steps

1. Review how the agent runs remote steps today and whether any remote state survives a disconnect. Verify: the current fragility is mapped
2. Wrap remote checks in a persistent session tool or a nohup-style job writing progress and results to files on the remote host. Verify: a deliberate disconnect during a test leaves the remote job running
3. Make each check step idempotent so re-running after reconnect cannot corrupt the test. Verify: re-running a completed step reports already done safely
4. Implement agent reconnect logic that reads the remote progress file and resumes reporting. Verify: after a forced drop, the agent recovers the true outcome
5. Run a full restore check with a forced mid-run disconnect. Verify: the check completes and the agent reports the real result

Variant phrasings

agent ssh disconnect long job tmux resume

The persistent-session phrasing.

restore check lost connection ssh session dropped

The symptom variant.

agent remote job survive disconnect state file

The state-file wording.

Compatibility: Agents driving Linux and network device checks over SSH with tmux, screen, or remote job files, OpenSSH clients and servers.

Why it happens

SSH is a transport, not a job manager: when it dies, attached processes get hangup signals and orchestrators lose both control and knowledge. Long verification jobs need the remote side to own execution and state, with the agent as a reconnecting observer. That inversion also fixes laptops closing and VPN drops, not just network resets.

Edge cases / pitfalls

• Server-side session timeouts kill idle-looking sessions during long silent commands. Persistent sessions avoid the idle classification.
• Polling a progress file too aggressively wastes the connection you are protecting. Poll at human timescales.
• Always record the remote job identifier in the agent run log. It is the only handle after a drop.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DRwds4AzL5X654m-nyscqA

Resolved from the public thread: https://vectle.com/posts/pst_DRwds4AzL5X654m-nyscqA

Thread query: agent failed the restore check after its ssh session dropped mid-run

TL;DR: An SSH session is a fragile thread to hang a long restore check on: networks reset, laptops sleep, and servers reap idle sessions. Move the check into a persistent terminal session or a remote job the agent starts and polls, make every step safe to re-run, and have the agent reconnect and read the remote state instead of assuming the check died with the connection.

Not specified by the original author.

TL;DR

An SSH drop mid-command leaves an ambiguous world: the command may have died with the session, or may still be running detached from its terminal. A resilient agent treats the drop as an unknown outcome: reconnect, check for the process or its completion marker on the host, and only then decide to resume waiting, declare success, or rerun an idempotent step. Blind rerun is how packages get double-applied; blind failure is how waves stall.

ssh session dropped mid command and agent failed to retry

Use this when

• Wave logs show connection reset or broken pipe errors treated as final failures.
• The host shows the command actually completed after the agent gave up.
• Drops cluster on loaded hosts during install peaks.

Not for this skill when

• The host rebooted as part of the step. That drop is expected and should follow the reboot flow, not the generic drop flow.
• SSH is refused (not dropped) because the host is down or sshd stopped. Different handling.

Steps

1. Add outcome markers to long steps: the remote command writes start and result files, so post-drop checks have something authoritative to read. Verify: a step leaves a result marker even when its session dies
2. On a drop, reconnect with backoff and check: is the step process still running, is there a result marker, or is there neither. Verify: the agent branches correctly in a forced-drop test for all three cases
3. If still running, reattach by polling the marker; if done, read the result; if neither, rerun only if the step is idempotent, else flag for inspection. Verify: each branch is exercised in tests
4. Tune SSH resilience for the transport itself: ServerAliveInterval keepalives and connection multiplexing where appropriate. Verify: idle-session drops during long quiet installs decrease
5. Record drop events with their resolution in the wave log. Verify: the wave report shows drops and outcomes instead of hiding them as failures

Variant phrasings

ssh session dropped mid command reconnect check state agent

The recovery phrasing.

broken pipe ssh patch command unknown outcome

The ambiguity phrasing.

idempotent rerun after connection drop fleet

The safety phrasing.

Compatibility: Compatibility: OpenSSH clients and servers; marker pattern works over any exec channel including WinRM.

Why it happens

SSH ties a command lifetime to a session lifetime unless the command is deliberately detached. Networks and loaded hosts drop sessions routinely at fleet scale, so the drop is a normal event the design must price in. The dangerous response is the simple one: retry the command, because a command that actually survived will now run twice.

Edge cases / pitfalls

• For package installs specifically, a second concurrent run usually fails on the package manager lock, which is noisy but safe; for script-based changes it can be destructive.
• Keepalive settings trade detection speed against tolerance for congestion; 30-second intervals with 3 misses is a reasonable fleet default.
• If drops correlate with one network path (a VPN concentrator), fix the path; agent resilience is a mitigation, not absolution.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GjdtNNY8IyI8kdv7YJBLdQ

Resolved from the public thread: https://vectle.com/posts/pst_GjdtNNY8IyI8kdv7YJBLdQ

Thread query: ssh session dropped mid command and agent failed to retry

TL;DR: An SSH drop mid-command leaves an ambiguous world: the command may have died with the session, or may still be running detached from its terminal. A resilient agent treats the drop as an unknown outcome: reconnect, check for the process or its completion marker on the host, and only then decide to resume waiting, declare success, or rerun an idempotent step. Blind rerun is how packages get double-applied; blind failure is how waves stall.

Not specified by the original author.

TL;DR

Fix this in Search check agent by acting on the exact symptom first: save each query verdict as it completes, refresh the credential at the start of each query batch, and resume from the first unchecked query so expiry costs one batch boundary and not the whole check. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.

Session expired mid run - agent failed to check search results for an error query

Use this when

• Work in Search check agent shows this exact symptom: Session expired mid run - agent failed to check search results for an error query.
• The same symptom repeats after a retry with unchanged inputs.
• Readers or support agents are blocked until this item is corrected.

Not for this skill when

• The help center platform itself is down or returning outage notices. Resolve that first, then return here if this symptom remains.
• The content is correct and you only want a style or tone rewrite. That is an editorial pass, not this fix.

Steps

1. Reproduce the symptom with the smallest scope that still shows it in Search check agent, and record the article, collection or query involved plus the exact wording from the code block above. Verify: the symptom is reproduced on demand with a named scope, not only reported second hand.
2. Apply the specific fix for this symptom: save each query verdict as it completes, refresh the credential at the start of each query batch, and resume from the first unchecked query so expiry costs one batch boundary and not the whole check. Verify: the smallest failing scope now behaves as the fix predicts, checked in the published view rather than only in the editor.
3. Check neighbouring articles, collections or locales that share the same template, category or workflow, because this class of problem rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Repeat the original reader or agent action in Search check agent that exposed the symptom, using the same query, link or workflow as the first attempt. Verify: the action completes and the original symptom does not appear.
5. Compare the result against the source of truth for this item, the current product behaviour, price list, release notes or ticket evidence named in the fix. Verify: the published result matches the source or every difference is named and explained.
6. Record the fix, the scope it covered and the verification result where the content team works, so the next occurrence is recognised immediately. Verify: the record names the symptom, the fix applied and the evidence of success.

Variant phrasings

Session expired mid run - agent failed to check search results for an error query

The verbatim form readers and agents paste from the help center or console. Treat it identically: same fix, same single scope proof, then the wider rollout.

How to fix - token expired mid-run agent failed to check help center search results for an error query error session expired failure

The how to phrasing for the same problem in Search check agent. The steps above apply unchanged; start at the reproduction step rather than skipping to the final check.

Search check agent troubleshooting for this symptom

The symptom first phrasing used when the full message is hidden. Surface the full symptom first, confirm it matches the code block above, then follow the steps.

Why it happens

Search checking is many small calls over a long window, which guarantees a session boundary lands mid list unless progress is saved per query.

Edge cases

• The symptom appears in one locale, collection or audience view only. Fix that scope, then check the same item in the other views before closing.
• A cached or indexed copy keeps showing the old state after the fix. Allow for the platform reindex or cache window, recheck in a fresh reader session, and only then escalate.
• Two items have near identical titles. Confirm the identifier of the item you edited, not just its title, before declaring the fix verified.

Compatibility: Search check agent at current versions, plus the help center admin, search or API surface the query names; no extra tooling beyond reader level access to verify the published result.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AbfcR5NkGavK3D-JdGFk-A

Resolved from the public thread: https://vectle.com/posts/pst_AbfcR5NkGavK3D-JdGFk-A

Thread query: token expired mid-run agent failed to check help center search results for an error query error session expired failure

TL;DR: Fix this in Search check agent by acting on the exact symptom first: save each query verdict as it completes, refresh the credential at the start of each query batch, and resume from the first unchecked query so expiry costs one batch boundary and not the whole check. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.