ssh host key changed mid run and agent failed verification
Fixes a patch agent failing SSH host key verification mid-run when a host key legitimately changed (host rebuilt, reimaged, or IP reused), and the agent either hard-fails or, worse, has verification disabled to cope. Use when sweeps stop on Host key verification failed for rebuilt hosts. Not for an actual unexpected key change on a host that was not rebuilt, which must be treated as a security event.
TL;DR
Host key changes mid-fleet-operation are routine in one case (rebuilds) and alarming in another (possible spoofing), and the agent must distinguish them. Source expected keys from the inventory or CMDB (recorded at provision time), verify against that, and when a change is expected because the CMDB shows a rebuild, update the known-hosts entry through that verified channel, never by blindly accepting on connect.
ssh host key changed mid run and agent failed verificationUse this when
- The agent stops with Host key verification failed on hosts that were just reprovisioned.
- Operators have started using StrictHostKeyChecking=no as the workaround.
- IP addresses are recycled in the environment, so keys and IPs do not stay paired.
Not for this skill when
- A key changed on a host with no rebuild record. Stop and treat it as a potential security incident.
- All hosts fail verification. The agent known-hosts file or its source is broken, not the fleet.
Steps
- Confirm the operational rule in force: automatic acceptance is forbidden; changes require an authoritative source. Verify: any existing accept-any configuration is found and scheduled for removal
- Wire host key sourcing to the inventory: at provisioning, record the host key fingerprint in the CMDB; the agent builds its known-hosts from that data. Verify: a new host first sweep verifies against the CMDB fingerprint
- For a failed host, check the CMDB for a rebuild or reimage event matching the change time. Verify: the change is explained by a recorded event, or it is escalated as unexplained
- Update the known-hosts entry for explained changes through the verified path and re-run the host. Verify: the host completes its sweep steps
- Add reporting: every key change, explained or not, lands in the wave report with its disposition. Verify: unexplained changes page a human the same run
Variant phrasings
host key verification failed rebuilt host agent cmdb fingerprint
The verified-path phrasing.
stricthostkeychecking no workaround risk fleet
The anti-pattern phrasing.
ssh host key changed ip reused cloud instances
The environment phrasing.
Compatibility: Compatibility: OpenSSH with known-hosts or a global known-hosts file generated from inventory data; some fleets use SSH certificates instead, which removes this class entirely.
Why it happens
SSH trust-on-first-use assumes a human who recognizes the host and notices surprises. Agents have neither property, so their host-key trust must come from data URIs the same system that told the agent the host exists should tell it the host key. Rebuild-heavy and IP-recycling environments make changes frequent, which is exactly when disabling verification feels tempting and is most dangerous.
Edge cases / pitfalls
- SSH host certificates (a CA signing host keys) are the structural fix for large fleets; the CMDB fingerprint approach is the pragmatic step before that.
- Keep the agent known-hosts generated, not hand-edited, or it drifts from the CMDB within weeks.
- An unexplained key change runbook should include out-of-band verification with the host owner before any acceptance.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstyEqdCI15DmQx8gODKnY4w