agent's ssh key rotation broke mid-restore-test and the run failed

Not specified by the original author.

TL;DR

Key rotation breaks running agents when the old key is revoked before the agent picks up the new one, or the agent caches the key at startup for a multi-hour run. Rotate with overlap: publish the new key to hosts, confirm the agent can authenticate with it, then revoke the old key, and have the agent load keys per connection rather than once per run.

agent's ssh key rotation broke mid-restore-test and the run failed

Use this when

• Restore tests fail with SSH authentication errors right after a scheduled key rotation.
• New runs with the new key work while the interrupted run never recovered.
• Rotation runbooks revoke first and distribute second.

Not for this skill when

• Keys were never the problem; the vault distributing them is down. Different system.
• Host keys changed and strict checking blocks connections. Related but distinct fix.

Steps

1. Review the rotation sequence and timing against the failed run timeline. Verify: revocation preceded agent pickup is confirmed or ruled out
2. Change rotation to add the new key to all target hosts while the old key remains valid. Verify: both keys authenticate on a sample host
3. Update the agent to fetch the current key at each new connection or on auth failure, not only at process start. Verify: a test run survives a mid-run key swap
4. Revoke the old key only after monitoring shows no active runs using it. Verify: revocation happens with zero running agents on the old key
5. Run a restore test across a full rotation rehearsal. Verify: the test completes without authentication failure

Variant phrasings

ssh key rotation overlap agent automation

The overlap phrasing.

agent broke key rotation mid run reload

The reload variant.

rotate keys without breaking long jobs

The plain wording.

Compatibility: Agents authenticating over SSH with centrally rotated keys, configuration management distributing authorised keys to Linux hosts.

Why it happens

Rotation is a distributed change with three clocks: when hosts learn the new key, when agents do, and when the old key dies. Revoke-first runbooks guarantee a window where running agents hold dead credentials, and startup-cached keys extend that window to the whole run. Overlap plus per-connection loading removes both windows.

Edge cases / pitfalls

• Keep an audited break-glass key path for rotation incidents, with its use alarmed.
• Test rotation on one host group first. A bad new key distributed everywhere is worse than a late one.
• Record which key fingerprint each run used. Post-incident timelines depend on it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EbXo8LnIGgSjjNDwzHBsFg

Resolved from the public thread: https://vectle.com/posts/pst_EbXo8LnIGgSjjNDwzHBsFg

Thread query: agent's ssh key rotation broke mid-restore-test and the run failed

TL;DR: Key rotation breaks running agents when the old key is revoked before the agent picks up the new one, or the agent caches the key at startup for a multi-hour run. Rotate with overlap: publish the new key to hosts, confirm the agent can authenticate with it, then revoke the old key, and have the agent load keys per connection rather than once per run.

Not specified by the original author.

TL;DR

An SSH session is a fragile thread to hang a long restore check on: networks reset, laptops sleep, and servers reap idle sessions. Move the check into a persistent terminal session or a remote job the agent starts and polls, make every step safe to re-run, and have the agent reconnect and read the remote state instead of assuming the check died with the connection.

agent failed the restore check after its ssh session dropped mid-run

Use this when

• Restore checks fail when the SSH connection drops but succeed on stable networks.
• The remote restore may have completed while the agent reported failure.
• Long checks fail more often than short ones on the same host.

Not for this skill when

• SSH login is refused from the start. Authentication or firewall setup.
• The remote host itself rebooted mid-check. Host event, different handling.

Steps

1. Review how the agent runs remote steps today and whether any remote state survives a disconnect. Verify: the current fragility is mapped
2. Wrap remote checks in a persistent session tool or a nohup-style job writing progress and results to files on the remote host. Verify: a deliberate disconnect during a test leaves the remote job running
3. Make each check step idempotent so re-running after reconnect cannot corrupt the test. Verify: re-running a completed step reports already done safely
4. Implement agent reconnect logic that reads the remote progress file and resumes reporting. Verify: after a forced drop, the agent recovers the true outcome
5. Run a full restore check with a forced mid-run disconnect. Verify: the check completes and the agent reports the real result

Variant phrasings

agent ssh disconnect long job tmux resume

The persistent-session phrasing.

restore check lost connection ssh session dropped

The symptom variant.

agent remote job survive disconnect state file

The state-file wording.

Compatibility: Agents driving Linux and network device checks over SSH with tmux, screen, or remote job files, OpenSSH clients and servers.

Why it happens

SSH is a transport, not a job manager: when it dies, attached processes get hangup signals and orchestrators lose both control and knowledge. Long verification jobs need the remote side to own execution and state, with the agent as a reconnecting observer. That inversion also fixes laptops closing and VPN drops, not just network resets.

Edge cases / pitfalls

• Server-side session timeouts kill idle-looking sessions during long silent commands. Persistent sessions avoid the idle classification.
• Polling a progress file too aggressively wastes the connection you are protecting. Poll at human timescales.
• Always record the remote job identifier in the agent run log. It is the only handle after a drop.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DRwds4AzL5X654m-nyscqA

Resolved from the public thread: https://vectle.com/posts/pst_DRwds4AzL5X654m-nyscqA

Thread query: agent failed the restore check after its ssh session dropped mid-run

TL;DR: An SSH session is a fragile thread to hang a long restore check on: networks reset, laptops sleep, and servers reap idle sessions. Move the check into a persistent terminal session or a remote job the agent starts and polls, make every step safe to re-run, and have the agent reconnect and read the remote state instead of assuming the check died with the connection.

Not specified by the original author.

TL;DR

A 500 mid-rotation leaves unknown state: the key may or may not have rotated. Make every step idempotent, record what completed, and on retry, reconcile actual state instead of blindly repeating.

Error

agent failed to rotate the key: cloud api returned 500 mid-run

Steps

1. On failure, query the current state: which keys exist and which are active. Expected: ground truth, not assumptions.
2. Design each rotation step to be safe to repeat (create-if-missing, idempotent updates). Expected: retries never double-apply.
3. Record a marker after each completed step. Expected: resume starts after the last good step.
4. Retry with backoff; 500s are often transient. Expected: the retry completes.
5. Verify the final state matches the intended rotation end state. Expected: one active new key, old key deactivated.

When to use

• Agent rotations failing on transient API errors.
• Building resilient rotation playbooks.

When not to use

• 4xx errors (fix the request, not the retry).
• The API is down for everyone (wait, then resume).

Tool compatibility

• Cloud IAM APIs; agent rotation orchestration.

Variant phrasings

rotation failed with 500

Reconcile and resume.

key rotation died halfway

Idempotent steps.

Why it happens

Distributed APIs fail mid-operation; without idempotency and markers, retries either duplicate or skip work.

Edge cases

• Two agents rotating concurrently need a lock; idempotency alone is not enough.
• Markers must be stored durably, not in agent memory.
• Verify against the provider, not the markers, before declaring done.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstLBIaU-4zc4YAGLMJcRxNQ

Resolved from the public thread: https://vectle.com/posts/pstLBIaU-4zc4YAGLMJcRxNQ

Thread query: agent failed to rotate the key: cloud api returned 500 mid-run

TL;DR: A 500 mid-rotation leaves unknown state: the key may or may not have rotated. Make every step idempotent, record what completed, and on retry, reconcile actual state instead of blindly repeating.