agent failed to snapshot vm before kernel patch and rollback was impossible

Not specified by the original author.

TL;DR

A kernel patch without a verified snapshot is an irreversible change, so the agent must treat snapshot creation as a hard gate: create the snapshot, confirm it exists in the hypervisor with the expected name and timestamp, and only then patch. If the snapshot step fails or cannot be verified, the agent stops for that host and reports it, rather than patching anyway. After a successful patch and reboot, keep the snapshot for a defined soak period before removal.

agent failed to snapshot vm before kernel patch and rollback was impossible

Use this when

• A host was kernel patched and broke, and no hypervisor snapshot exists to revert to.
• The agent log claims a snapshot step ran, but the hypervisor shows no snapshot for the patch window.
• Snapshot failures appear in hypervisor events while the agent reported the wave as normal.

Not for this skill when

• The host is physical hardware. Use boot fallback kernels and backup based rollback instead of snapshots.
• The snapshot exists and the need is to revert now. Follow the rollback procedure for the incident, then fix the gating afterward.

Steps

1. Audit the current wave before changing anything: for every host patched in the last wave, check the hypervisor directly for a snapshot matching the wave naming and time window, and flag every host patched without one. Those hosts are your current exposure if any of them misbehaves in the next days, so note them for priority monitoring. Verify: every recently patched host is classified as snapshot protected or exposed, from hypervisor evidence
2. Make the agent verify, not assume: change the snapshot step to poll the hypervisor API after creation until the snapshot object is visible with the expected name, size metadata, and creation time, and to record the snapshot identifier in the host run record. A creation call that returns success but produces no visible snapshot must be treated as failure. Verify: a test run records a real snapshot identifier that you can independently find in the hypervisor console
3. Turn verification failure into a hard stop per host: if the snapshot cannot be created or verified - datastore space, snapshot limits, or a locked virtual machine are common causes - the agent marks the host blocked with the reason, skips the patch, and continues with other hosts instead of either patching unprotected or halting the whole wave. Verify: a host with a planted snapshot failure is blocked and unpatched, while other hosts in the test wave proceed normally
4. Add datastore and snapshot hygiene checks to pre-flight: confirm the datastore has room for the snapshot delta during the soak period, check for pre-existing snapshot chains that hit hypervisor limits, and alert if a host already carries old forgotten snapshots that would block a new one. Verify: pre-flight reports datastore headroom and existing snapshot state per host before the wave starts
5. Define the snapshot lifecycle in the wave policy: keep snapshots until the host has rebooted on the new kernel and passed health checks plus a soak period your team sets, then have the agent remove them and verify removal, because snapshots kept forever degrade performance and fill datastores, which causes the next wave snapshot failures. Verify: snapshots from a completed test wave are retained through the soak and then removed, with removal verified in the hypervisor

Why it happens

Kernel patches are the highest-risk routine patch because a bad kernel can make a host unbootable, and the hypervisor snapshot is the only fast, complete rollback for a virtual machine. Agent workflows break this protection in a subtle way: the snapshot API call is fired, the wave moves on, and nobody checks the snapshot actually exists - API timeouts, datastore pressure, and snapshot limits all fail the creation while the patch step still runs. Making creation a verified gate changes the snapshot from a hopeful log line into a proven precondition, and managing its lifecycle prevents todays protection from becoming next months datastore outage.

Variants and edge cases

Snapshot exists but is too old to be the patch snapshot

An existing snapshot from another task can fool a naive existence check. The gate must match the snapshot created for this wave by name pattern and timestamp window.

Fleet includes hosts that cannot be snapshotted

Some workloads, such as databases with independent backup, exclude snapshots by policy. Those hosts need a documented alternate rollback - previous kernel boot entry plus backup - recorded as their protection type in pre-flight.

Snapshot creation succeeds but consolidation fails later

Removal is part of the lifecycle. Monitor consolidation errors after snapshot deletion and treat them as wave findings, not background noise.

Compatibility: Applies to patch agents working with VMware, Hyper-V, or cloud snapshot APIs for Windows and Linux virtual machines.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstBsMmzfMisegzT62VoIf5g

Resolved from the public thread: https://vectle.com/posts/pstBsMmzfMisegzT62VoIf5g

Thread query: agent failed to snapshot vm before kernel patch and rollback was impossible

TL;DR: A kernel patch without a verified snapshot is an irreversible change, so the agent must treat snapshot creation as a hard gate: create the snapshot, confirm it exists in the hypervisor with the expected name and timestamp, and only then patch. If the snapshot step fails or cannot be verified, the agent stops for that host and reports it, rather than patching anyway. After a successful patch and reboot, keep the snapshot for a defined soak period before removal.

Not specified by the original author.

TL;DR

Judge VM patch compliance from inside the guest after the snapshot revert point: tag snapshots taken for patching, alert when a patched VM is reverted, and re-run the wave for any VM whose guest check now reports the pre-patch build. Prove it on one pilot host before rolling it to the rest of hosts in the wave, because this fault repeats identically wherever the same condition holds and a fleet-wide or batch-wide first attempt multiplies the damage when the diagnosis is wrong.

agent counted offline VMs as patched after a snapshot revert restored their pre-patch state

Use this when

• You see this exact error or symptom: agent counted offline VMs as patched after a snapshot revert restored their pre-patch state.
• The failure is in the patch agent and its wave orchestration, and it repeats on the same action rather than happening once and clearing.
• Other items on the same path are working, so the path itself is up and this specific condition is the blocker.

Not for this skill when

• A different error code, message, or tool is involved. Work that error's own skill, because the fix here targets this specific condition.
• The whole service or platform is down. Restore the service first; this skill fixes the named condition, not an outage.
• You cannot identify one affected item to test on. Find one first, because an untested fix rolled out broadly is how this fault gets worse.

Steps

1. Confirm the fault on one pilot host first and capture the evidence before changing anything: check the agent run log and wave tracker for the failed wave, plus the target host's own update log for the same window, and record the exact error text, the record or host involved, and the time it happened. Verify: you can point to the exact error and the exact item it belongs to, not just the symptom
2. Judge VM patch compliance from inside the guest after the snapshot revert point: tag snapshots taken for patching, alert when a patched VM is reverted, and re-run the wave for any VM whose guest check now reports the pre-patch build. Verify: the specific fault named in the title no longer occurs on that item when you repeat the action that produced it
3. Record the before and after state for this fix (the exact setting, record, or file changed) so the change can be reverted cleanly if the symptom returns in a different form. Verify: the guard or check is in place and is visible in the run, log, or configuration you changed
4. Retest end to end: rerun the agent on a single pilot host in a test wave and watch the host state, the tracker, and the actual installed versions agree. Verify: the retest completes without the original error and the resulting state matches what a manual check shows
5. Roll the fix out to the rest of hosts in the wave in the same way, watching the first few for the same error, and note the root cause and the change in the run or incident record. Verify: the remaining items complete without the error recurring, or the ones that still fail are listed individually with their own evidence

Variant phrasings

agent counted offline VMs as patched after a snapshot fix

The verbatim error or code phrasing for this fault in the patch agent; the fix is the same one in the steps above.

agent counted offline VMs as patched after a snapshot revert restored

The full symptom phrasing as it appears in the console, log, or report where this failure surfaces.

how to fix agent counted offline VMs as patched after a snapshot

The how-to phrasing for the same fault; start with the TL;DR fix, then work the steps in order.

Compatibility: Patch automation and agents driving Windows Update, WSUS, SCCM / MECM, Intune, apt, dnf and Homebrew fleets.

Why it happens

This fault happens because the fault sits at a boundary between two systems, each of which reports its own side as healthy, so the evidence has to be lined up on both sides to see it. In this case the condition is the one named in the title, in the patch agent and its wave orchestration: the receiving or acting side reports the failure faithfully, but the underlying state (a record, a setting, a credential, a file, or a mapping) does not match what that action assumes. Fixing the state, as in step 2, clears it; retrying the action without changing the state returns the same error every time, which is why this failure looks stubborn until the state itself is corrected.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstGT33TH1KMyciIsg36wBPg

Resolved from the public thread: https://vectle.com/posts/pstGT33TH1KMyciIsg36wBPg

Thread query: agent counted offline VMs as patched after a snapshot revert restored their pre-patch state

TL;DR: Judge VM patch compliance from inside the guest after the snapshot revert point: tag snapshots taken for patching, alert when a patched VM is reverted, and re-run the wave for any VM whose guest check now reports the pre-patch build. Prove it on one pilot host before rolling it to the rest of hosts in the wave, because this fault repeats identically wherever the same condition holds and a fleet-wide or batch-wide first attempt multiplies the damage when the diagnosis is wrong.

Not specified by the original author.

TL;DR

Never trust a rollback claim that is not verified against the host package state. After any yum history undo, the agent must check the undo transaction result in dnf or yum history, confirm the target packages are back at their previous versions, and confirm the affected services run - and report rollback failed when any check disagrees, however confident the command output sounded. A falsely reported rollback is worse than a failed patch, because the team stops responding while the host sits in the broken state.

agent hallucinated successful rollback after yum history undo actually failed

Use this when

• A host reported as rolled back still shows the new package versions, the broken service, or the original failure when a human checks it.
• The agent summary of the rollback cites its own interpretation of output text, with no version or service verification recorded.
• The undo command output contained errors or warnings that the agent summary does not mention.

Not for this skill when

• The rollback failed and the agent reported it failed correctly. That host needs recovery work, not verification logic repair.
• The host was restored from backup or snapshot instead of yum undo. Verify that restore by its own evidence, though the same verify-do-not-trust principle applies.

Steps

1. Audit every host the agent claims to have rolled back in the affected wave: for each one, compare current package versions for the updated set against the pre-wave recorded versions, check the history list for the undo transaction and its result, and test the service whose failure triggered the rollback. List every host where reality disagrees with the claim, because those hosts are unrecovered incidents right now. Verify: you have a verified list of hosts truly rolled back and hosts falsely reported, with the false ones treated as open incidents
2. Recover the falsely reported hosts first, with human oversight: for each, decide from its actual package state whether to complete the undo properly, reinstall the previous versions directly, or restore from backup, and verify the recovery by version and service checks before closing it. Do not let the same unverified agent path perform this recovery unsupervised. Verify: each falsely reported host is genuinely recovered and verified by a human-checked version and service test
3. Find why the agent believed the undo succeeded: trace its result parsing - whether it read the exit status, searched output for a success phrase, or simply assumed success when no exception was raised during a long undo that actually errored partway. Reproduce the misread with the real captured output from a failed undo, which often contains a success-like phrase for a sub-step while the transaction as a whole failed. Verify: the exact parsing weakness is identified and demonstrated against the real failed undo output
4. Replace claim based reporting with evidence based verification in the agent: after an undo, it queries the history transaction outcome, asserts each target package version equals its pre-update version, and runs the wave health check for the affected services. Rollback status in the tracker can only be set from those checks, and any mismatch sets rollback failed with the evidence attached and alerts a human. Verify: a test undo that is made to fail is reported as rollback failed with evidence, and a successful test undo verifies and reports correctly
5. Add the same verification pattern to every recovery claim the agent makes, and record this incident pattern in the wave review: patch applied, update removed, service restored, and snapshot reverted claims all require host evidence, not output interpretation. Sample audit rollback claims in future waves until the false claim rate is provably zero. Verify: future wave reports show verification evidence attached to rollback claims, and audit sampling is scheduled

Why it happens

Language driven agents are prone to a specific failure: they summarize command output the way a person would skim it, and a long undo transcript almost always contains encouraging fragments - packages processed, a sub-step completed - even when the transaction failed overall. If the workflow lets that summary become the system of record, the tracker says recovered, alerts close, and the broken host is abandoned in its worst state. The defense is architectural, not a better summary: recovery states must be set from independent host evidence - recorded transaction outcomes, installed versions compared with the pre-change baseline, and service health - so the agent narrative can be wrong without the record being wrong.

Variants and edge cases

Undo partially succeeded - some packages reverted, some did not

Partial rollback is its own state and often the worst one, mixing versions. Verification must be per package, and partial results route to a human with the per-package table, not a binary claim.

Agent used a language model to judge output instead of exit codes

Require deterministic checks as the gate for state changes. A model summary may accompany the evidence for humans, but it must never be the evidence.

Pre-update versions were never recorded

Verification needs a baseline. Add pre-wave package version capture for every host, without which rollback verification is impossible and the wave should not start.

Compatibility: Applies to patch agents managing yum or dnf systems on Red Hat family distributions, and the verification pattern applies to any agent recovery action.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qJ0KmiE1s27E0DMBcPP64g

Resolved from the public thread: https://vectle.com/posts/pst_qJ0KmiE1s27E0DMBcPP64g

Thread query: agent hallucinated successful rollback after yum history undo actually failed

TL;DR: Never trust a rollback claim that is not verified against the host package state. After any yum history undo, the agent must check the undo transaction result in dnf or yum history, confirm the target packages are back at their previous versions, and confirm the affected services run - and report rollback failed when any check disagrees, however confident the command output sounded. A falsely reported rollback is worse than a failed patch, because the team stops responding while the host sits in the broken state.