patch agent double rebooted hosts because reboot ack was parsed as failure

Not specified by the original author.

TL;DR

A host going down for its planned reboot looks, to a naive agent, exactly like a connection failure. If the agent parses the dropped session or a terse acknowledgment as failure, it retries the reboot command when the host returns - rebooting it again. Fix the acknowledgment handling: treat the expected disconnect after issuing a reboot as a planned transition, wait for the host to go down and come back with a new boot time, and mark reboot complete from that evidence. A reboot command must be issued exactly once per host per wave unless a human decides otherwise.

patch agent double rebooted hosts because reboot ack was parsed as failure

Use this when

• Hosts reboot twice during one wave, with the second reboot shortly after the first completes.
• The agent log marks the first reboot as failed or timed out, then shows a retry that succeeded.
• Users or services complain of an unexpected second outage in the same window.

Not for this skill when

• A specific update legitimately requires a second reboot to finish configuration and the agent planned both. That should be documented as a two stage reboot, not a parsing bug.
• The host itself rebooted spontaneously due to a crash. Investigate the crash, not the agent parser.

Steps

1. Prove the sequence from both sides: align the agent log timestamps for one host with the host own event log or journal showing its shutdown initiators, and with its boot times. Confirm the agent issued the reboot command twice, and that the first host shutdown matches the first command, which separates agent double issuance from a host that rebooted itself. Verify: agent and host logs together show two agent-issued reboots for one host in one wave
2. Find the parsing fault precisely: read how the agent interprets the reboot command response - the remote shell exit behavior, the session drop, and any acknowledgment text - and reproduce it against a test host or a recorded response. Note the exact signal it misread, such as a non-zero exit from a closing connection or an acknowledgment string in a different format after an agent or host tooling update changed it. Verify: the misread signal is identified and reproduced, with the response the agent received documented
3. Rebuild reboot completion on host evidence instead of command response: after issuing a reboot once, the agent records the pre-reboot boot time, marks the host rebooting, waits for the host to become unreachable, then waits for it to return and compares boot time. Only a changed boot time plus passing health checks completes the state, and no path in this flow reissues the reboot command automatically. Verify: a test host completes its reboot state from boot time evidence with exactly one reboot command in the agent log
4. Make the reboot command itself idempotent in effect: before issuing any reboot, the agent checks the host current state in the tracker and on the host - if a reboot is already recorded as issued or the host boot time already advanced past the install, it skips issuing another. This guard protects against agent restarts and retries elsewhere in the workflow re-entering the reboot step. Verify: re-entering the reboot step on an already rebooted test host produces a skip decision and no command
5. Regression test the acknowledgment variants: cover the response formats your fleet actually produces across its Windows and Linux tooling - clean acknowledgment, dropped session, slow shutdown, and a host that fails to reboot at all - and assert the agent issues one command and reaches the right end state for each. The genuine failure case must end as an alert, never as an automatic second reboot. Verify: the test set passes for all variants, including the no-reboot case ending in an alert with no retry command

Why it happens

Rebooting is the one patching action whose success destroys the channel used to observe it: the remote session drops because the host is doing exactly what it was told. Agents that judge actions by their command response therefore face an ambiguity - a dropped connection can mean the reboot started, or that the command never arrived. Parsing an acknowledgment string makes it worse, because acknowledgment formats vary by tooling version and locale. The robust resolution is to stop judging the reboot by its command at all and judge it by the host state transition: unreachable, then reachable with a new boot time. That evidence is unambiguous, survives agent restarts, and makes a second automatic reboot structurally impossible.

Variants and edge cases

Double reboot only on one remote tooling path

If SSH managed hosts are fine but WinRM hosts double reboot, the acknowledgment handling differs per transport. Fix and test each transport path separately rather than changing the shared logic blindly.

Host takes so long to shut down the agent assumes failure

Long shutdowns from services that stop slowly mimic a failed reboot. The wait for unreachable must have a generous, measured ceiling and end in an alert, not a retry.

Second reboot was actually the update own restart

Some updates restart the host themselves mid-install. The agent state model must attribute host initiated restarts during install to the install phase, so its own planned reboot afterward is still exactly one.

Compatibility: Applies to patch agents using SSH, WinRM, or management tooling to reboot Windows and Linux hosts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0J3HQZM5rIXYLsf90DMQdA

Resolved from the public thread: https://vectle.com/posts/pst_0J3HQZM5rIXYLsf90DMQdA

Thread query: patch agent double rebooted hosts because reboot ack was parsed as failure

TL;DR: A host going down for its planned reboot looks, to a naive agent, exactly like a connection failure. If the agent parses the dropped session or a terse acknowledgment as failure, it retries the reboot command when the host returns - rebooting it again. Fix the acknowledgment handling: treat the expected disconnect after issuing a reboot as a planned transition, wait for the host to go down and come back with a new boot time, and mark reboot complete from that evidence. A reboot command must be issued exactly once per host per wave unless a human decides otherwise.

Not specified by the original author.

TL;DR

Reboot pending is a success-in-progress state, not a failure flavor. The agent state model must separate install outcome from reboot state: a host can be installed-and-pending-reboot, and its final verdict waits for the post-reboot verification. Fix the classification where the flag is read (the compliance query or the agent status mapping), align the report categories with the model, and make retries skip pending-reboot hosts until they have rebooted and been rechecked.

agent failed by treating reboot pending flag as update failure

Use this when

• Failure counts include hosts that show the update as installed after a manual check and a reboot.
• The agent retries installs on hosts that are simply waiting for their reboot window.
• Failure rate drops to near zero after someone reboots the flagged hosts by hand.

Not for this skill when

• The host needs a reboot because an earlier, different update failed and rolled back to a pending state. Read the per-update state, not just the flag.
• The flag is stale and never clears (a servicing flag bug on the host). That host needs the stale-flag remediation, not just reclassification.

Steps

1. Trace where the agent reads host state and how the reboot-pending flag maps into its outcome enum. Verify: the exact mapping that converts pending into failed is located
2. Change the state model: install outcome (installed, failed, not attempted) tracked separately from reboot state (none, pending, done), with final success requiring installed plus reboot done plus post-reboot verification. Verify: the model is reflected in the tracker schema and the report
3. Update retry logic to exclude installed-pending-reboot hosts from reinstall attempts and route them to the reboot phase. Verify: a test wave performs zero reinstalls on pending hosts
4. Reclassify the current fleet records using host evidence, moving false failures to their true states. Verify: the corrected failure rate is restated in the wave report with a note
5. Add a report check: pending-reboot hosts older than a defined age (for example past their reboot window) escalate as a reboot-scheduling problem, not as patch failures. Verify: aging pending hosts surface under the right problem label

Variant phrasings

reboot pending flag counted as update failure agent state model

The main phrasing.

installed pending reboot separate outcome and reboot state

The modeling phrasing.

false failure rate patch report pending reboot hosts

The reporting phrasing.

Compatibility: Compatibility: Windows hosts (CBS and Windows Update pending flags) and Linux (reboot-required indicators such as the var run reboot-required file on Debian-family).

Why it happens

The bug is a modeling shortcut: one flag read as one verdict. Servicing reality is two-dimensional, because installing and rebooting are separate acts separated by policy and time, especially in fleets where reboots wait for windows. Collapsing the dimensions manufactures failures, and manufactured failures trigger real harm: reinstall attempts on healthy hosts, wasted windows, and a failure metric nobody trusts, which then hides genuine failures in the noise.

Edge cases / pitfalls

• Keep the dimensions separate in every downstream consumer (dashboards, alerts, SLA math), not only in the tracker, or the shortcut reappears in reporting.
• A host can be pending reboot from an update outside the wave; the model should record which update set the flag when that is knowable.
• Post-reboot verification closes the loop: pending is a promise, verification is the proof.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstV9pvfiAoDWSjycW6iBR9Q

Resolved from the public thread: https://vectle.com/posts/pstV9pvfiAoDWSjycW6iBR9Q

Thread query: agent failed by treating reboot pending flag as update failure

TL;DR: Reboot pending is a success-in-progress state, not a failure flavor. The agent state model must separate install outcome from reboot state: a host can be installed-and-pending-reboot, and its final verdict waits for the post-reboot verification. Fix the classification where the flag is read (the compliance query or the agent status mapping), align the report categories with the model, and make retries skip pending-reboot hosts until they have rebooted and been rechecked.

Not specified by the original author.

TL;DR

A reboot-ack timeout is usually a timing model problem: the coordinator expects the host back in a fixed short window, but hosts with many services, disk checks, or firmware updates take far longer, and the agent own probe may target a service that starts late. Replace the single timeout with staged readiness: ping, then SSH or WinRM, then the patch agent or service check, over a generous total budget, and record hosts as reboot-pending rather than failed while they are inside that budget.

patch coordinator agent failed on reboot-ack timeout

Use this when

• The coordinator marks hosts failed at the reboot step, but they are up and healthy when checked.
• Large database or storage hosts fail this step more than small VMs.
• The timeout value is a single flat number for every host class.

Not for this skill when

• The host truly never returns and needs console recovery. That is a boot incident, not an ack timeout.
• The coordinator never sent the reboot because the pre-reboot command failed. Different step, different error.

Steps

1. Measure real reboot durations per host class from logs (reboot issued to SSH answering). Verify: you have a duration distribution, not a guess
2. Replace the single ack wait with staged probes: network reachable, remote exec answering, then patch-relevant service up, each with its own sub-timeout inside one overall budget based on the measured worst case plus headroom. Verify: the staged probe logic is in the coordinator config or code
3. Change the state model so a host inside its reboot budget is reboot-pending, excluded from failure counts and from duplicate reboot commands. Verify: the wave tracker shows the pending state distinctly
4. Add a hard ceiling (for example 60 minutes) after which the host escalates to a human with console instructions. Verify: a simulated hung host escalates rather than waiting forever
5. Run the next wave and compare reboot-step failures. Verify: reboot-ack failures drop to only genuinely stuck hosts

Variant phrasings

reboot acknowledgement timeout patch coordinator staged readiness

The fix-pattern phrasing.

agent gives up waiting host reboot slow host returns later

The symptom phrasing.

reboot pending state vs failed state wave tracker

The state-model phrasing.

Compatibility: Compatibility: Coordinator agents using SSH, WinRM, or cloud APIs for reboot orchestration; timings must be tuned per fleet.

Why it happens

Reboot time is dominated by things the patch did not change: firmware POST, filesystem checks on large volumes, and service start ordering. A coordinator written against laptop-class reboot times will systematically fail exactly the biggest, most important servers. Staged probes also fix a subtler bug: probing the patch service first makes a host look dead while it is simply still starting services.

Edge cases / pitfalls

• Firmware updates bundled with some patches extend reboot by 10 minutes or more; detect that case from the patch metadata and extend the budget automatically.
• Never issue a second reboot to a host in reboot-pending state; a double reboot mid-service-start creates the failures you are trying to avoid.
• Record actual reboot durations per host; the fleet timing model should learn from every wave.

Provenance

Resolved from the public thread: https://vectle.com/posts/psttHVs9tyJfnDg0OaRgnemg

Resolved from the public thread: https://vectle.com/posts/psttHVs9tyJfnDg0OaRgnemg

Thread query: patch coordinator agent failed on reboot-ack timeout

TL;DR: A reboot-ack timeout is usually a timing model problem: the coordinator expects the host back in a fixed short window, but hosts with many services, disk checks, or firmware updates take far longer, and the agent own probe may target a service that starts late. Replace the single timeout with staged readiness: ping, then SSH or WinRM, then the patch agent or service check, over a generous total budget, and record hosts as reboot-pending rather than failed while they are inside that budget.