agent hallucinated successful rollback after yum history undo actually failed

TL;DR

Never trust a rollback claim that is not verified against the host package state. After any yum history undo, the agent must check the undo transaction result in dnf or yum history, confirm the target packages are back at their previous versions, and confirm the affected services run - and report rollback failed when any check disagrees, however confident the command output sounded. A falsely reported rollback is worse than a failed patch, because the team stops responding while the host sits in the broken state.

agent hallucinated successful rollback after yum history undo actually failed

Use this when

• A host reported as rolled back still shows the new package versions, the broken service, or the original failure when a human checks it.
• The agent summary of the rollback cites its own interpretation of output text, with no version or service verification recorded.
• The undo command output contained errors or warnings that the agent summary does not mention.

Not for this skill when

• The rollback failed and the agent reported it failed correctly. That host needs recovery work, not verification logic repair.
• The host was restored from backup or snapshot instead of yum undo. Verify that restore by its own evidence, though the same verify-do-not-trust principle applies.

Steps

1. Audit every host the agent claims to have rolled back in the affected wave: for each one, compare current package versions for the updated set against the pre-wave recorded versions, check the history list for the undo transaction and its result, and test the service whose failure triggered the rollback. List every host where reality disagrees with the claim, because those hosts are unrecovered incidents right now. Verify: you have a verified list of hosts truly rolled back and hosts falsely reported, with the false ones treated as open incidents
2. Recover the falsely reported hosts first, with human oversight: for each, decide from its actual package state whether to complete the undo properly, reinstall the previous versions directly, or restore from backup, and verify the recovery by version and service checks before closing it. Do not let the same unverified agent path perform this recovery unsupervised. Verify: each falsely reported host is genuinely recovered and verified by a human-checked version and service test
3. Find why the agent believed the undo succeeded: trace its result parsing - whether it read the exit status, searched output for a success phrase, or simply assumed success when no exception was raised during a long undo that actually errored partway. Reproduce the misread with the real captured output from a failed undo, which often contains a success-like phrase for a sub-step while the transaction as a whole failed. Verify: the exact parsing weakness is identified and demonstrated against the real failed undo output
4. Replace claim based reporting with evidence based verification in the agent: after an undo, it queries the history transaction outcome, asserts each target package version equals its pre-update version, and runs the wave health check for the affected services. Rollback status in the tracker can only be set from those checks, and any mismatch sets rollback failed with the evidence attached and alerts a human. Verify: a test undo that is made to fail is reported as rollback failed with evidence, and a successful test undo verifies and reports correctly
5. Add the same verification pattern to every recovery claim the agent makes, and record this incident pattern in the wave review: patch applied, update removed, service restored, and snapshot reverted claims all require host evidence, not output interpretation. Sample audit rollback claims in future waves until the false claim rate is provably zero. Verify: future wave reports show verification evidence attached to rollback claims, and audit sampling is scheduled

Why it happens

Language driven agents are prone to a specific failure: they summarize command output the way a person would skim it, and a long undo transcript almost always contains encouraging fragments - packages processed, a sub-step completed - even when the transaction failed overall. If the workflow lets that summary become the system of record, the tracker says recovered, alerts close, and the broken host is abandoned in its worst state. The defense is architectural, not a better summary: recovery states must be set from independent host evidence - recorded transaction outcomes, installed versions compared with the pre-change baseline, and service health - so the agent narrative can be wrong without the record being wrong.

Variants and edge cases

Undo partially succeeded - some packages reverted, some did not

Partial rollback is its own state and often the worst one, mixing versions. Verification must be per package, and partial results route to a human with the per-package table, not a binary claim.

Agent used a language model to judge output instead of exit codes

Require deterministic checks as the gate for state changes. A model summary may accompany the evidence for humans, but it must never be the evidence.

Pre-update versions were never recorded

Verification needs a baseline. Add pre-wave package version capture for every host, without which rollback verification is impossible and the wave should not start.

Compatibility: Applies to patch agents managing yum or dnf systems on Red Hat family distributions, and the verification pattern applies to any agent recovery action.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qJ0KmiE1s27E0DMBcPP64g