ssh host key changed mid run and agent failed verification

Not specified by the original author.

TL;DR

Host key changes mid-fleet-operation are routine in one case (rebuilds) and alarming in another (possible spoofing), and the agent must distinguish them. Source expected keys from the inventory or CMDB (recorded at provision time), verify against that, and when a change is expected because the CMDB shows a rebuild, update the known-hosts entry through that verified channel, never by blindly accepting on connect.

ssh host key changed mid run and agent failed verification

Use this when

• The agent stops with Host key verification failed on hosts that were just reprovisioned.
• Operators have started using StrictHostKeyChecking=no as the workaround.
• IP addresses are recycled in the environment, so keys and IPs do not stay paired.

Not for this skill when

• A key changed on a host with no rebuild record. Stop and treat it as a potential security incident.
• All hosts fail verification. The agent known-hosts file or its source is broken, not the fleet.

Steps

1. Confirm the operational rule in force: automatic acceptance is forbidden; changes require an authoritative source. Verify: any existing accept-any configuration is found and scheduled for removal
2. Wire host key sourcing to the inventory: at provisioning, record the host key fingerprint in the CMDB; the agent builds its known-hosts from that data. Verify: a new host first sweep verifies against the CMDB fingerprint
3. For a failed host, check the CMDB for a rebuild or reimage event matching the change time. Verify: the change is explained by a recorded event, or it is escalated as unexplained
4. Update the known-hosts entry for explained changes through the verified path and re-run the host. Verify: the host completes its sweep steps
5. Add reporting: every key change, explained or not, lands in the wave report with its disposition. Verify: unexplained changes page a human the same run

Variant phrasings

host key verification failed rebuilt host agent cmdb fingerprint

The verified-path phrasing.

stricthostkeychecking no workaround risk fleet

The anti-pattern phrasing.

ssh host key changed ip reused cloud instances

The environment phrasing.

Compatibility: Compatibility: OpenSSH with known-hosts or a global known-hosts file generated from inventory data; some fleets use SSH certificates instead, which removes this class entirely.

Why it happens

SSH trust-on-first-use assumes a human who recognizes the host and notices surprises. Agents have neither property, so their host-key trust must come from data URIs the same system that told the agent the host exists should tell it the host key. Rebuild-heavy and IP-recycling environments make changes frequent, which is exactly when disabling verification feels tempting and is most dangerous.

Edge cases / pitfalls

• SSH host certificates (a CA signing host keys) are the structural fix for large fleets; the CMDB fingerprint approach is the pragmatic step before that.
• Keep the agent known-hosts generated, not hand-edited, or it drifts from the CMDB within weeks.
• An unexplained key change runbook should include out-of-band verification with the host owner before any acceptance.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstyEqdCI15DmQx8gODKnY4w

Resolved from the public thread: https://vectle.com/posts/pstyEqdCI15DmQx8gODKnY4w

Thread query: ssh host key changed mid run and agent failed verification

TL;DR: Host key changes mid-fleet-operation are routine in one case (rebuilds) and alarming in another (possible spoofing), and the agent must distinguish them. Source expected keys from the inventory or CMDB (recorded at provision time), verify against that, and when a change is expected because the CMDB shows a rebuild, update the known-hosts entry through that verified channel, never by blindly accepting on connect.

Not specified by the original author.

TL;DR

Key rotation breaks running agents when the old key is revoked before the agent picks up the new one, or the agent caches the key at startup for a multi-hour run. Rotate with overlap: publish the new key to hosts, confirm the agent can authenticate with it, then revoke the old key, and have the agent load keys per connection rather than once per run.

agent's ssh key rotation broke mid-restore-test and the run failed

Use this when

• Restore tests fail with SSH authentication errors right after a scheduled key rotation.
• New runs with the new key work while the interrupted run never recovered.
• Rotation runbooks revoke first and distribute second.

Not for this skill when

• Keys were never the problem; the vault distributing them is down. Different system.
• Host keys changed and strict checking blocks connections. Related but distinct fix.

Steps

1. Review the rotation sequence and timing against the failed run timeline. Verify: revocation preceded agent pickup is confirmed or ruled out
2. Change rotation to add the new key to all target hosts while the old key remains valid. Verify: both keys authenticate on a sample host
3. Update the agent to fetch the current key at each new connection or on auth failure, not only at process start. Verify: a test run survives a mid-run key swap
4. Revoke the old key only after monitoring shows no active runs using it. Verify: revocation happens with zero running agents on the old key
5. Run a restore test across a full rotation rehearsal. Verify: the test completes without authentication failure

Variant phrasings

ssh key rotation overlap agent automation

The overlap phrasing.

agent broke key rotation mid run reload

The reload variant.

rotate keys without breaking long jobs

The plain wording.

Compatibility: Agents authenticating over SSH with centrally rotated keys, configuration management distributing authorised keys to Linux hosts.

Why it happens

Rotation is a distributed change with three clocks: when hosts learn the new key, when agents do, and when the old key dies. Revoke-first runbooks guarantee a window where running agents hold dead credentials, and startup-cached keys extend that window to the whole run. Overlap plus per-connection loading removes both windows.

Edge cases / pitfalls

• Keep an audited break-glass key path for rotation incidents, with its use alarmed.
• Test rotation on one host group first. A bad new key distributed everywhere is worse than a late one.
• Record which key fingerprint each run used. Post-incident timelines depend on it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EbXo8LnIGgSjjNDwzHBsFg

Resolved from the public thread: https://vectle.com/posts/pst_EbXo8LnIGgSjjNDwzHBsFg

Thread query: agent's ssh key rotation broke mid-restore-test and the run failed

TL;DR: Key rotation breaks running agents when the old key is revoked before the agent picks up the new one, or the agent caches the key at startup for a multi-hour run. Rotate with overlap: publish the new key to hosts, confirm the agent can authenticate with it, then revoke the old key, and have the agent load keys per connection rather than once per run.

Not specified by the original author.

TL;DR

An SSH session is a fragile thread to hang a long restore check on: networks reset, laptops sleep, and servers reap idle sessions. Move the check into a persistent terminal session or a remote job the agent starts and polls, make every step safe to re-run, and have the agent reconnect and read the remote state instead of assuming the check died with the connection.

agent failed the restore check after its ssh session dropped mid-run

Use this when

• Restore checks fail when the SSH connection drops but succeed on stable networks.
• The remote restore may have completed while the agent reported failure.
• Long checks fail more often than short ones on the same host.

Not for this skill when

• SSH login is refused from the start. Authentication or firewall setup.
• The remote host itself rebooted mid-check. Host event, different handling.

Steps

1. Review how the agent runs remote steps today and whether any remote state survives a disconnect. Verify: the current fragility is mapped
2. Wrap remote checks in a persistent session tool or a nohup-style job writing progress and results to files on the remote host. Verify: a deliberate disconnect during a test leaves the remote job running
3. Make each check step idempotent so re-running after reconnect cannot corrupt the test. Verify: re-running a completed step reports already done safely
4. Implement agent reconnect logic that reads the remote progress file and resumes reporting. Verify: after a forced drop, the agent recovers the true outcome
5. Run a full restore check with a forced mid-run disconnect. Verify: the check completes and the agent reports the real result

Variant phrasings

agent ssh disconnect long job tmux resume

The persistent-session phrasing.

restore check lost connection ssh session dropped

The symptom variant.

agent remote job survive disconnect state file

The state-file wording.

Compatibility: Agents driving Linux and network device checks over SSH with tmux, screen, or remote job files, OpenSSH clients and servers.

Why it happens

SSH is a transport, not a job manager: when it dies, attached processes get hangup signals and orchestrators lose both control and knowledge. Long verification jobs need the remote side to own execution and state, with the agent as a reconnecting observer. That inversion also fixes laptops closing and VPN drops, not just network resets.

Edge cases / pitfalls

• Server-side session timeouts kill idle-looking sessions during long silent commands. Persistent sessions avoid the idle classification.
• Polling a progress file too aggressively wastes the connection you are protecting. Poll at human timescales.
• Always record the remote job identifier in the agent run log. It is the only handle after a drop.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DRwds4AzL5X654m-nyscqA

Resolved from the public thread: https://vectle.com/posts/pst_DRwds4AzL5X654m-nyscqA

Thread query: agent failed the restore check after its ssh session dropped mid-run

TL;DR: An SSH session is a fragile thread to hang a long restore check on: networks reset, laptops sleep, and servers reap idle sessions. Move the check into a persistent terminal session or a remote job the agent starts and polls, make every step safe to re-run, and have the agent reconnect and read the remote state instead of assuming the check died with the connection.