agent failed when patch window closed mid transaction and dnf left

Not specified by the original author.

TL;DR

A dnf transaction interrupted mid-flight can leave duplicate, half-installed, or unconfigured packages, so do not start a new update on top of it. Recover in order: check dnf history and the transaction state, run the distribution recovery steps - complete or roll back the interrupted transaction, remove duplicates, and rebuild the rpm database if it complains - then run a full check before resuming patching. Change the agent so window ends stop new transactions from starting and let a running transaction finish or reach a safe checkpoint.

agent failed when patch window closed mid transaction and dnf left system half upgraded

Use this when

• A host shows duplicate packages, failed transaction messages, or inconsistent rpm state after a patch window closed during a dnf run.
• The agent log shows the transaction was killed, the session timed out, or the host lost power mid transaction.
• dnf check or a test transaction reports problems referencing the interrupted run.

Not for this skill when

• The dnf transaction completed cleanly and one package service fails afterward. Fix that service, not the transaction state.
• The window closed before any transaction started. There is nothing to recover; simply reschedule.

Steps

1. Freeze further patching on the host and gather the transaction evidence: read dnf history for the interrupted transaction and its state, check for any still running package processes, and confirm no other administrator session is mid transaction. Record the transaction identifier and the package count it touched, because recovery decisions depend on how far it got. Verify: patching is paused on the host and the interrupted transaction is identified with its scope documented
2. Assess consistency with the built-in checks before changing anything: run dnf check and an rpm database verification, and note duplicate packages, unsatisfied dependencies, or database errors. Duplicates are the signature of an interrupted install where the new version installed but the old version cleanup never ran. Verify: the checks produce a concrete list of duplicates and inconsistencies, or a database problem that must be repaired first
3. Recover the transaction using the distribution supported path: if dnf offers to complete or redo the transaction, take it; use dnf history undo or redo for the interrupted transaction where appropriate, and remove exact duplicate older packages with the duplicate cleanup tooling your distribution provides, such as package-cleanup equivalents or dnf remove duplicates functionality. Work on a console session that will not be timed out again. Verify: the interrupted transaction is completed or undone, duplicates are removed, and dnf check returns clean or a much smaller, specific problem
4. Rebuild trust in the package state: run a full dnf distro sync or update dry run to see the resolver view of the system, complete any pending configuration, and reboot if kernel or core libraries were part of the interrupted set, since half-updated libraries can leave running processes on deleted files. Verify: a dry run shows a coherent package set and the host reboots cleanly if core components were involved
5. Fix the agent window behavior so this cannot recur: the agent must check remaining window time against the estimated transaction duration before starting, must never kill a transaction at window end but instead let it finish while blocking new work, and must record a transaction in progress state durably so a restarted agent recovers it instead of starting a second transaction on top. Verify: a simulated window close during a test transaction lets the transaction finish safely and the agent records and recovers its state correctly

Why it happens

RPM transactions are designed to be atomic in intent but not in execution: packages are installed, configured, and old versions erased in sequence, and an interruption between those phases leaves the database describing a system that is part old and part new, including the classic duplicate package state where the erase never happened. Starting another transaction on that foundation compounds the inconsistency because the resolver trusts the database. Recovery tools work because they reason about the recorded transaction as a whole. The agent lesson is scheduling discipline: a maintenance window is a deadline for starting work, never a license to kill package transactions mid-flight.

Variants and edge cases

Interruption was a host power loss or hypervisor snapshot revert

Treat database integrity as suspect: verify the rpm database, and if the host was reverted, reconcile its package state against the tracker, which may believe updates the revert undid.

Duplicates involve the kernel

Kernel duplicates are often normal because multiple kernels coexist by design. Exclude kernel packages from duplicate panic and use the installonly packages configuration to judge them.

Agent window is enforced by an external killer

If infrastructure force-kills sessions at window end, the agent must start transactions only when the full estimated duration fits, with margin, and checkpoint its intent durably before starting.

Compatibility: Applies to dnf based systems including Red Hat Enterprise Linux 8 and 9, Rocky Linux, AlmaLinux, and Fedora.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gRkxtgaoveuR-Cxrg2Cm7Q

Resolved from the public thread: https://vectle.com/posts/pst_gRkxtgaoveuR-Cxrg2Cm7Q

Thread query: agent failed when patch window closed mid transaction and dnf left system half upgraded

TL;DR: A dnf transaction interrupted mid-flight can leave duplicate, half-installed, or unconfigured packages, so do not start a new update on top of it. Recover in order: check dnf history and the transaction state, run the distribution recovery steps - complete or roll back the interrupted transaction, remove duplicates, and rebuild the rpm database if it complains - then run a full check before resuming patching. Change the agent so window ends stop new transactions from starting and let a running transaction finish or reach a safe checkpoint.

Not specified by the original author.

TL;DR

A clean abort is a designed behavior, not an accident: the agent must know the window deadline, stop starting new targets before it, let in-flight installs reach a safe checkpoint (never kill a package manager mid-transaction), mark each host state precisely (patched, in-progress-safe, untouched), and hand the remainder to the next window.

agent failed to abort cleanly when patch window overran

Use this when

• After an overrun, some hosts have half-configured packages or pending operations nobody recorded.
• The agent was killed by an operator or a scheduler timeout at window end.
• The next wave has no reliable starting picture.

Not for this skill when

• The wave finished but reporting was lost. That is a reporting bug, not an abort-behavior bug.
• A single host hung its install indefinitely. That is a per-host hang; abort design assumes installs eventually checkpoint.

Steps

1. Define the window contract in the coordinator: a soft deadline after which no new targets start, sized to the longest single-target patch time. Verify: the deadline and buffer are explicit config values
2. Implement graceful stop: on deadline or abort signal, the agent stops dequeuing, signals in-flight steps to finish the current package transaction, and waits a bounded grace period. Verify: a test abort never kills dpkg, rpm, or the Windows installer mid-transaction
3. Write per-host terminal states for the wave (complete, safe-partial with the exact step, untouched) to the tracker before exiting. Verify: after a test abort, every target has a state and a next action
4. Generate the carry-over list automatically: untouched and safe-partial hosts become the head of the next window queue. Verify: the next wave plan starts with the carry-over list
5. Review the abort in the wave report: how many completed, carried over, and why the wave overran (estimate error versus host slowness). Verify: the report drives a sizing change for the next window

Variant phrasings

patch window overrun graceful abort agent design

The design phrasing.

never kill package manager mid transaction abort

The safety-rule phrasing.

carry over list next maintenance window patching

The continuity phrasing.

Compatibility: Compatibility: Any coordinator with a scheduler or window concept; grace mechanics differ by package manager but the state model is portable.

Why it happens

Patch windows are hard external constraints (business hours return whether the wave is done or not), while package transactions are unsafe to interrupt. An agent without an abort design resolves that tension in the worst way: it either runs past the window or dies mid-transaction. The fix is to treat the deadline as an input the planner respects, not a wall the executor crashes into.

Edge cases / pitfalls

• Size waves from measured per-host patch times, not averages across unlike hosts; the tail is what overruns windows.
• Some updates cannot checkpoint mid-way (firmware, certain drivers); the planner should schedule those first in the window, never last.
• An abort that leaves a host safe-partial must also leave a note of which reboot or step remains, or the next operator guesses.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstyQZEb_rn4yzg1O0BBPH9A

Resolved from the public thread: https://vectle.com/posts/pstyQZEb_rn4yzg1O0BBPH9A

Thread query: agent failed to abort cleanly when patch window overran

TL;DR: A clean abort is a designed behavior, not an accident: the agent must know the window deadline, stop starting new targets before it, let in-flight installs reach a safe checkpoint (never kill a package manager mid-transaction), mark each host state precisely (patched, in-progress-safe, untouched), and hand the remainder to the next window.

Not specified by the original author.

TL;DR

False success is the worst patch-agent bug because it poisons every downstream decision. The rule: success means verified state, not a happy exit code. After install, the agent must re-query the host (is the KB listed as installed, is the package version the target version, did a post-reboot check pass) and classify outcomes as installed, installed-pending-reboot, failed, or unknown. Know the treacherous codes: Windows 3010 means success plus reboot required, while some installers return 0 for a rollback.

agent reported success on hosts where the update install failed

Use this when

• Audits or vulnerability scans find missing patches on hosts reported patched.
• The agent success rate is implausibly perfect across waves.
• Success was recorded at install time with no post-check.

Not for this skill when

• The patch installed and verified, then a later update or admin removed it. That is drift after success, handled by reconciliation.
• The host is mid-reboot and genuinely unknowable at report time. It should be unknown, not success.

Steps

1. List the outcome signals the agent currently trusts and find where verification is absent. Verify: the gap (exit code only, log scrape only) is pinpointed
2. Implement post-install verification per platform: query installed updates or package version, compare against the target set, and record the raw evidence. Verify: verification output is stored per host, not just a boolean
3. Handle reboot-required outcomes as a distinct state and verify again after the reboot completes. Verify: 3010-class results are not counted as final success until the post-reboot check passes
4. Treat verification failure or unreachable-after-install as failed or unknown respectively, never success. Verify: a test with a sabotaged install yields a non-success state
5. Backfill: run verification across the last wave reported successes and report the true rate. Verify: the corrected success rate is published in the wave record

Variant phrasings

agent reported success but patch failed verify installed state

The core phrasing.

exit code 3010 success reboot required misread

The treacherous-code phrasing.

false success rate patch audit verification step

The audit phrasing.

Compatibility: Compatibility: Windows (installed update queries, DISM, WMI) and Linux (rpm, dpkg version queries); verifier design is platform-specific, the state model is not.

Why it happens

Installers are optimistic reporters: wrappers swallow errors, exit codes are overloaded, and Windows servicing in particular defers the real verdict until after a reboot completes the transaction. An agent that records the installer opinion instead of the host state will manufacture false confidence at scale, and false confidence in patching becomes unpatched vulnerabilities nobody is looking for.

Edge cases / pitfalls

• Sample-verify even when full verification is expensive; a 10 percent audit sample catches systematic false success early.
• Keep the distinction between failed and unknown sharp; merging them hides unreachable hosts inside the failure count.
• When a vendor installer is a known liar, wrap it: snapshot state before, install, diff state after, and judge from the diff.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_c9qiMfelx-dKZWQMuVFQIg

Resolved from the public thread: https://vectle.com/posts/pst_c9qiMfelx-dKZWQMuVFQIg

Thread query: agent reported success on hosts where the update install failed

TL;DR: False success is the worst patch-agent bug because it poisons every downstream decision. The rule: success means verified state, not a happy exit code. After install, the agent must re-query the host (is the KB listed as installed, is the package version the target version, did a post-reboot check pass) and classify outcomes as installed, installed-pending-reboot, failed, or unknown. Know the treacherous codes: Windows 3010 means success plus reboot required, while some installers return 0 for a rollback.