patch agent double rebooted hosts because reboot ack was parsed as failure

TL;DR

A host going down for its planned reboot looks, to a naive agent, exactly like a connection failure. If the agent parses the dropped session or a terse acknowledgment as failure, it retries the reboot command when the host returns - rebooting it again. Fix the acknowledgment handling: treat the expected disconnect after issuing a reboot as a planned transition, wait for the host to go down and come back with a new boot time, and mark reboot complete from that evidence. A reboot command must be issued exactly once per host per wave unless a human decides otherwise.

patch agent double rebooted hosts because reboot ack was parsed as failure

Use this when

• Hosts reboot twice during one wave, with the second reboot shortly after the first completes.
• The agent log marks the first reboot as failed or timed out, then shows a retry that succeeded.
• Users or services complain of an unexpected second outage in the same window.

Not for this skill when

• A specific update legitimately requires a second reboot to finish configuration and the agent planned both. That should be documented as a two stage reboot, not a parsing bug.
• The host itself rebooted spontaneously due to a crash. Investigate the crash, not the agent parser.

Steps

1. Prove the sequence from both sides: align the agent log timestamps for one host with the host own event log or journal showing its shutdown initiators, and with its boot times. Confirm the agent issued the reboot command twice, and that the first host shutdown matches the first command, which separates agent double issuance from a host that rebooted itself. Verify: agent and host logs together show two agent-issued reboots for one host in one wave
2. Find the parsing fault precisely: read how the agent interprets the reboot command response - the remote shell exit behavior, the session drop, and any acknowledgment text - and reproduce it against a test host or a recorded response. Note the exact signal it misread, such as a non-zero exit from a closing connection or an acknowledgment string in a different format after an agent or host tooling update changed it. Verify: the misread signal is identified and reproduced, with the response the agent received documented
3. Rebuild reboot completion on host evidence instead of command response: after issuing a reboot once, the agent records the pre-reboot boot time, marks the host rebooting, waits for the host to become unreachable, then waits for it to return and compares boot time. Only a changed boot time plus passing health checks completes the state, and no path in this flow reissues the reboot command automatically. Verify: a test host completes its reboot state from boot time evidence with exactly one reboot command in the agent log
4. Make the reboot command itself idempotent in effect: before issuing any reboot, the agent checks the host current state in the tracker and on the host - if a reboot is already recorded as issued or the host boot time already advanced past the install, it skips issuing another. This guard protects against agent restarts and retries elsewhere in the workflow re-entering the reboot step. Verify: re-entering the reboot step on an already rebooted test host produces a skip decision and no command
5. Regression test the acknowledgment variants: cover the response formats your fleet actually produces across its Windows and Linux tooling - clean acknowledgment, dropped session, slow shutdown, and a host that fails to reboot at all - and assert the agent issues one command and reaches the right end state for each. The genuine failure case must end as an alert, never as an automatic second reboot. Verify: the test set passes for all variants, including the no-reboot case ending in an alert with no retry command

Why it happens

Rebooting is the one patching action whose success destroys the channel used to observe it: the remote session drops because the host is doing exactly what it was told. Agents that judge actions by their command response therefore face an ambiguity - a dropped connection can mean the reboot started, or that the command never arrived. Parsing an acknowledgment string makes it worse, because acknowledgment formats vary by tooling version and locale. The robust resolution is to stop judging the reboot by its command at all and judge it by the host state transition: unreachable, then reachable with a new boot time. That evidence is unambiguous, survives agent restarts, and makes a second automatic reboot structurally impossible.

Variants and edge cases

Double reboot only on one remote tooling path

If SSH managed hosts are fine but WinRM hosts double reboot, the acknowledgment handling differs per transport. Fix and test each transport path separately rather than changing the shared logic blindly.

Host takes so long to shut down the agent assumes failure

Long shutdowns from services that stop slowly mimic a failed reboot. The wait for unreachable must have a generous, measured ceiling and end in an alert, not a retry.

Second reboot was actually the update own restart

Some updates restart the host themselves mid-install. The agent state model must attribute host initiated restarts during install to the install phase, so its own planned reboot afterward is still exactly one.

Compatibility: Applies to patch agents using SSH, WinRM, or management tooling to reboot Windows and Linux hosts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0J3HQZM5rIXYLsf90DMQdA