## TL;DR
A patch wave needs a circuit breaker: define an error rate threshold over a meaningful sample - for example failures as a share of attempted hosts once a minimum number have been tried - and have the agent check it after every host, pause the wave automatically when it trips, and alert with the evidence. A wave that keeps installing through a systemic failure turns a bad update or a broken source into a fleet incident. Test the breaker with a simulated failure burst before trusting it in production.

```text
agent failed to pause wave when error rate crossed threshold mid deployment
```

## Use this when
- A wave continued to most of the fleet even though early hosts were failing at a high rate.
- Post-incident review shows the failure pattern was visible in the first portion of the wave.
- The agent has retry and per-host error handling but no wave-level stop condition.

## Not for this skill when
- Failures are sporadic and stay well below any systemic level across a full wave. Per-host handling is the right level there.
- The wave was deliberately run to completion for an emergency patch with a human accepting the risk in real time. Document that decision rather than retrofitting a breaker trip.

## Steps
1. Reconstruct the incident wave numerically: from the run records, compute the failure rate after every ten attempted hosts and find the point where a reasonable threshold would have tripped. This grounds the threshold in your own fleet data instead of a copied number, and shows how much of the fleet the breaker would have saved. Verify: you have the failure curve of the incident wave and the trip point a proposed threshold would have produced
2. Define the breaker policy explicitly: a minimum sample size before the rate is judged, the failure rate that trips, which error classes count - systemic classes such as source, package, and store errors count immediately, while host-specific classes such as offline may be excluded - and the exact actions on trip: stop starting new hosts, let in-flight installs finish safely, preserve state, and alert. Verify: the policy is written down with sample size, rate, counted error classes, and trip actions, in the run configuration rather than only in documentation
3. Implement the check in the wave loop, not as a reporting afterthought: after each host completes, update the rolling counters and evaluate the breaker before dispatching the next host. Log every evaluation near the boundary so a later review can see the wave approaching the trip, not just the trip itself. Verify: code inspection and a dry run show the breaker evaluated between hosts with its counters logged
4. Test it destructively in a lab wave: point a test wave at a deliberately broken source or package so failures accumulate, and confirm the agent pauses at the threshold, does not start further hosts, sends the alert with the failure summary, and can be resumed by a human after the cause is fixed without losing completed host records. Verify: the lab wave trips the breaker at the expected point, alerts correctly, and resumes cleanly after a simulated fix
5. Set the resume and override rules: resuming requires a human acknowledgment that names the cause found, and any override to run through a tripped breaker is time-boxed and recorded on the wave. Review the threshold after each real wave for the first few cycles and tune it if normal waves come close to tripping. Verify: resume and override actions in a test leave a named human decision on the wave record, and the threshold review is scheduled

## Why it happens
Per-host error handling answers the question can this host be patched; a circuit breaker answers should this wave continue at all. Systemic failures - a corrupt package in the source, a bad update, a datastore outage - present as a rising failure rate across otherwise unrelated hosts, and an agent without a wave-level view will faithfully convert that signal into fleet-wide damage at machine speed, far faster than a human watching a console. A threshold breaker works because systemic causes reveal themselves early in a staged wave; stopping at a defined rate caps the blast radius at the sample size plus a margin, which is exactly the protection staging was designed to give.

## Variants and edge cases
### Emergency zero-day patch where speed matters more
Lower the minimum sample and monitor in real time with a human watching, rather than disabling the breaker. Even emergency waves benefit from a trip that catches a broken package.

### Failure rate spikes from one host class only
Scope the breaker per class or ring as well as globally, so a broken server model pauses its own wave segment without stopping healthy populations.

### Breaker trips on a transient source outage that self-recovers
Require the rate to persist across a short recheck of the source health before pausing installs permanently, but still stop new starts while the recheck runs.

Compatibility: Applies to custom patch orchestration agents and to staged deployments in Configuration Manager, Ansible, and similar tooling where wave logic can be extended.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SVLOQx5cwvK4Swu0b1hVaw