patch agent lost track of which hosts had rebooted after orchestrator restart

TL;DR

Reboot state must live in durable storage, not in the orchestrator process. Persist every host state transition - installed, reboot scheduled, reboot detected, verified - to the tracker before the agent acts on it, and on startup have the agent reconcile: read the tracker, then verify each open host against reality by checking boot time and update state. A restart then costs a reconciliation pass, not the wave memory, and no host gets rebooted twice or verified without ever rebooting.

patch agent lost track of which hosts had rebooted after orchestrator restart

Use this when

• The orchestrator restarted mid-wave and afterward could not say which hosts had rebooted.
• Hosts were rebooted a second time, or marked verified, because the agent re-derived state from scratch after a restart.
• The tracker and the agent in-memory state disagree after any unplanned agent stop.

Not for this skill when

• The agent never restarted and reboot tracking failed for a different reason, such as hosts not reporting boot time. Fix the reporting path.
• A single host reboot state is wrong while the system generally recovers fine. Correct that host record and sample others before redesigning.

Steps

1. Reconstruct the affected wave from ground truth: for every host in the interrupted wave, determine its real state from the host itself - installed updates, current boot time compared with the install time, and pending reboot flags - and rebuild the tracker from that evidence. Do not trust either the pre-crash tracker or the agent post-restart guesses over host evidence. Verify: the tracker now matches host-verified reality for every host in the interrupted wave, with discrepancies listed
2. Identify every state the agent held only in memory: walk the wave workflow and list each transition, timer, and pending action, then check which of them were written to durable storage before the agent relied on them. The reboot detection wait is the classic gap, but approval holds and retry counts often share it. Verify: you have a list of in-memory-only states, with reboot tracking confirmed among them
3. Persist transitions before acting: change the workflow so a host state change is written to the tracker first, including intended actions such as reboot scheduled for a time, and only then executed. On execution, record the observed result - for reboots, the boot time the agent expects to beat - so recovery can tell whether the action happened while the agent was down. Verify: a code review and a test run show tracker writes preceding actions, with expected boot markers recorded for scheduled reboots
4. Build the startup reconciliation: when the agent starts and finds a wave in progress, it loads open host states, queries each host current boot time and update state, and advances or corrects each record - reboot detected if boot time passed the marker, still pending if not, and escalated if the host is unreachable beyond a threshold. Only after reconciliation does the wave continue dispatching new work. Verify: killing and restarting the agent mid test wave produces a reconciliation report and no host is double rebooted or wrongly verified
5. Test the failure you actually fear: run a drill wave against test hosts, kill the orchestrator at a random point, restart it, and diff the final wave outcome against an uninterrupted control run. Repeat the drill whenever the wave state model changes, and keep the drill script with the agent tests. Verify: the drill passes with identical final host states between interrupted and control runs

Why it happens

Orchestrators are processes, and processes restart - for patches of their own, host maintenance, crashes, and deploys. Any wave state held only in process memory is therefore state the design has decided to lose. Reboot tracking is the most fragile example because it spans an intentional disconnect: the host disappears, reboots on its own schedule, and the proof it happened is a changed boot time that must be compared against a remembered expectation. If that expectation lived in memory, the restarted agent cannot distinguish rebooted from not-yet-rebooted hosts and guesses, causing double reboots or false verification. Durable transitions plus startup reconciliation make restarts a routine event instead of an incident.

Variants and edge cases

Agent runs as several replicas

Multiple writers make durable state harder: use a single tracker with proper locking or a database, and ensure only one replica owns a wave through a lease, or reconciliation will fight itself.

Hosts are unreachable during reconciliation

Unreachable is a state, not an error to guess through. Park unreachable hosts with a retry schedule and alert only when they exceed the wave deadline.

Tracker itself was lost in the restart

If the tracker lived on the same host and storage as the agent and both were lost, rebuild from host ground truth as in step one, and move the tracker to independent durable storage afterward.

Compatibility: Applies to custom patch orchestrators for Windows and Linux fleets, using file, database, or service based trackers.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZZf4MXDXFshNiM5xDeIpRQ