agent's key rotation runbook failed halfway: partial rotation recovery

agent's key rotation runbook failed halfway: partial rotation recovery

TL;DR

The runbook died after rotating some consumers but not all, leaving a mix of old and new keys. Do not start over with a newer key; that creates a third version to track. Record exactly which consumers have which version, restore dual-accept (both keys valid) where needed so nothing is broken right now, fix whatever killed the runbook, then resume the same runbook from the failed step with the same new key.

agent's key rotation runbook failed halfway: partial rotation recovery

Use this when

• A rotation runbook errored partway and consumers are split across versions
• Some services authenticate and some do not after a rotation attempt
• Re-running the runbook from the top would mint yet another new key
• The runbook has no checkpoint and you need to recover manually

Not for this skill when

• Both old and new keys are rejected everywhere (that is a double-rotation, restore from version history first)
• The key was leaked (that is incident response: revoke everything, then rotate)
• You are designing rotation for the first time

Steps

1. Inventory every consumer: figure out which key version each one is actually using by checking configs and watching auth attempts. Expected: a complete list mapping each consumer to old or new.

2. Re-enable the old key anywhere it was disabled prematurely, so every consumer authenticates again right now. Expected: all consumers working; the incident is stopped even though rotation is incomplete.

3. Fix whatever killed the runbook: the timeout, the permission, the bad template. Expected: the root cause is addressed, not just the symptom.

4. Re-run the runbook from the failed step with the same new key, verifying each consumer before moving to the next. Expected: all consumers on the new key, each verified.

5. Only then disable the old key. Expected: the old key is retired with no consumer left depending on it.

Variant: secret rotation script died halfway through

Same recovery. Add per-step state to the script before the next run so a retry resumes instead of restarting.

Variant: partial key rotation some services still on old key

The inventory in step 1 is the whole job; most of the time the "stuck" services are the ones nobody thought to check.

Variant: rotation runbook failed resume from checkpoint

If the runbook kept no checkpoint, the inventory you build in step 1 becomes the checkpoint. Write it down; resume from it.

Why this happens

Rotation is a multi-step distributed change across consumers that do not move in lockstep. Without per-step state and idempotent steps, any mid-run failure (timeout, permission error, bad config) leaves consumers split across versions, and a blind retry either duplicates work or mints conflicting new keys.

Edge cases and pitfalls

• Make each step idempotent so re-running it is safe; a step that mints a new key on every run is the bug.
• Keep the old key valid until the very end (the dual-accept window); retiring it early is what turns a partial rotation into an outage.
• Log which version each consumer used at each step; the next recovery starts from that log.
• Test the runbook against staging first, including killing it halfway on purpose to prove resume works.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstW6QgjYo3bT82Kbtg5MYMQ