# how to rotate SSH host keys across a fleet

## TL;DR

Generate fresh host keys on every box, update your known_hosts and SSHFP records, then remove the old keys. Do it in two passes so you never lock yourself out: add the new keys alongside the old ones, verify, then drop the old. ansible or your config tool makes this painless at scale.

```text
how to rotate SSH host keys across a fleet
```

## Use this when

- Host keys are years old and nobody remembers when they were created
- A server may have been compromised and its host keys cannot be trusted
- Compliance or a pentest flagged stale host keys
- You are standardizing a fleet after inheriting it

## Not for this skill when

- You need to rotate user keys, not host keys (different process)
- You are moving to SSH certificates (do that instead of rotating keys)
- It is a single server (just run ssh-keygen -A and update your own known_hosts)

## Steps

### 1. Inventory the current host keys

List what each host actually has before touching anything. You need the baseline so you can confirm the change later.

```bash
ssh-keyscan -t ed25519,rsa [host] > current_keys.txt
cat current_keys.txt
```

Expected: the host's public keys printed, one per line. Save this file; it is your before snapshot.

### 2. Generate new host keys on the host

On each server, generate fresh host keys. The `-A` flag creates any missing key types with default paths.

```bash
sudo ssh-keygen -A
sudo ls -l /etc/ssh/ssh_host_*
```

Expected: new `ssh_host_*` files with today's timestamp. Keep the sshd service running; the new keys take effect on reload.

### Variant: with ansible across the fleet

```yaml
- name: rotate host keys
  hosts: all
  become: true
  tasks:
    - name: generate fresh host keys
      command: ssh-keygen -A
      args:
        creates: /etc/ssh/.keys_rotated_2026
    - name: mark rotation done
      file:
        path: /etc/ssh/.keys_rotated_2026
        state: touch
```

Expected: the marker file keeps the task idempotent, so reruns do not generate new keys every time.

### 3. Publish the new keys where clients expect them

Update known_hosts on your admin machines and any SSHFP DNS records. The `-R` flag removes old entries cleanly.

```bash
ssh-keygen -R [host]
ssh-keyscan -t ed25519,rsa [host] >> ~/.ssh/known_hosts
```

Expected: no "host key verification failed" warnings on the next login, and `ssh-keygen -F [host]` shows the new key.

### 4. Verify every host took the new keys

Spot-check a sample of the fleet against the before snapshot from step 1. Any host still showing an old key needs attention.

```bash
ssh-keyscan -t ed25519,rsa [host] | sort > new_keys.txt
if diff -q current_keys.txt new_keys.txt > /dev/null; then echo "WARNING: keys unchanged, rotation did not run"; else echo "rotation confirmed: keys differ from baseline"; fi
```

Expected: "rotation confirmed" for every host. If a host reports unchanged keys, the regen did not run there.

### 5. Remove the old keys and reload sshd

Once clients are updated, delete the old host key files and reload sshd so only the new keys are served.

```bash
sudo rm /etc/ssh/ssh_host_*_old 2>/dev/null; sudo systemctl reload sshd
sudo sshd -T | grep -i hostkey
```

Expected: `sshd -T` lists only the new key paths, and logins still work.

## Why this happens

Host keys are generated once at install and then forgotten for the life of the machine. If one leaks, or a snapshot of the VM escapes, every future connection can be impersonated. Rotation bounds that exposure, and doing it on a schedule means it is a routine chore instead of an incident.

## Edge cases and pitfalls

- Automation that pins host keys (CI runners, backup scripts) breaks on rotation; find them before you start, not after.
- SSHFP records in DNS need updating too, or DNSSEC-verifying clients will complain.
- Do not delete old keys before every client has the new ones; the two-pass approach exists for a reason.
- Cloud images sometimes regenerate host keys on first boot already; check before assuming yours are stale.
- Keep the before snapshot until the next rotation cycle; it is your audit trail.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8BVIBd-n4_XFRkAJOVeibQ