patch agent failed to expire its own maintenance window lock after crashing mid wave

TL;DR

A maintenance lock without a heartbeat is a future outage: when the agent crashes, the lock stays and every later wave refuses to run. Give locks an owner, a heartbeat, and an expiry: the agent refreshes the lock while it works, other runs may take over a lock whose heartbeat is older than your threshold after confirming no wave process is actually running, and every takeover is logged. On startup, the agent must check for and reconcile stale locks before concluding that patching is blocked.

patch agent failed to expire its own maintenance window lock after crashing mid wave

Use this when

• Patch waves report a maintenance lock held, but no agent process or wave is running.
• The lock dates from a crashed or killed agent run and was never released.
• Operators have been deleting lock files by hand as an undocumented routine.

Not for this skill when

• A wave is genuinely in progress and holding its lock. Wait for it or stop it deliberately; never treat a live lock as stale.
• The lock is held by a different system, such as a backup or a hypervisor operation. Coordinate with that owner instead of expiring their lock.

Steps

1. Clear the current stale lock safely, once, by hand: confirm from process lists, wave records, and host activity that no patch wave is running anywhere that could own the lock, record the lock contents - owner, time, wave - for the incident record, and only then remove or release it. Run one small wave immediately to prove patching works again before declaring the incident over. Verify: the lock is released after a documented no-owner check, and a test wave starts and completes normally
2. Redesign the lock as a lease, not a flag: the lock record stores the owning run identifier, the host or scope it covers, a heartbeat timestamp the agent updates at a fixed interval while working, and an expiry duration. Creating the replacement lock must be atomic in whatever store you use - file creation semantics, a database record, or a service call - so two agents cannot both believe they hold it. Verify: the new lock implementation writes owner, heartbeat, and expiry atomically, shown in a test with two competing agent starts
3. Add stale takeover with proof of death: a starting agent that finds a lock checks the heartbeat age and independently confirms the owner is gone - no matching process, no run record activity, and where possible a direct check of the owning host or orchestrator - before taking over. The takeover writes a new lock naming the previous owner and the evidence used, and sends an alert so a human sees that a crash left state behind. Verify: a simulated crashed run lock is taken over only after its heartbeat expires and owner absence is confirmed, with the takeover logged and alerted
4. Make the agent release its lock on every exit path it controls: normal completion, handled errors, and shutdown signals all release the lock in a finally style cleanup, and a wave that ends with hosts unfinished records their state before releasing, so releasing the lock never means losing the wave progress. The lease expiry then only has to cover the truly uncontrolled exits, such as a host power loss. Verify: test runs ending in success, handled failure, and a graceful stop all release the lock, and their host states are intact in the tracker
5. Audit for other locks with the same flaw and set an operational check: search the agent and its tooling for other lock files, mutex records, and maintenance flags set without expiry - snapshot locks, tracker locks, and per-host maintenance marks are common - and apply the same lease pattern. Add a daily check that reports any lock older than its expiry so stale locks are found by monitoring before they block a wave. Verify: other locks are inventoried and converted or documented, and a monitoring check reports expired locks with none silently blocking waves

Why it happens

Locks coordinate exclusive work, but a lock is a claim that must be continuously justified by a living owner. Simple flag locks - create a file at wave start, delete it at the end - assume the owner always reaches the delete, which crashes, kills, and power losses falsify. The result is a system that fails safe in the worst direction: permanently blocked, with no signal except waves that mysteriously never run. Lease based locks with heartbeats are the standard distributed systems answer because they encode the difference between held by a worker and abandoned by a crash, and the takeover evidence trail turns each recovery into a visible crash report instead of a silent manual file deletion.

Variants and edge cases

Lock lives in a shared tracker used by several agent replicas

Use the store transactional features for lock acquisition and heartbeat updates, and ensure the takeover check reads the same store the owner writes, not a cached copy.

Maintenance lock is per host, not per wave

Stale per-host locks strand individual hosts out of every wave. The lease pattern applies per host, and the startup reconciliation should sweep for expired host locks across the fleet.

Operators fear automatic takeover could double-run a wave

Pair takeover with the wave state reconciliation: a taking-over agent resumes from durable host states rather than starting fresh, so even a mistaken takeover continues work instead of duplicating it.

Compatibility: Applies to custom patch agents using file locks, database records, or coordination services, on Windows and Linux orchestrators.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstDJ7HQImdoiaH2XhZhpYtg