agent's key rotation runbook failed halfway: partial rotation recovery

Not specified by the original author.

agent's key rotation runbook failed halfway: partial rotation recovery

TL;DR

The runbook died after rotating some consumers but not all, leaving a mix of old and new keys. Do not start over with a newer key; that creates a third version to track. Record exactly which consumers have which version, restore dual-accept (both keys valid) where needed so nothing is broken right now, fix whatever killed the runbook, then resume the same runbook from the failed step with the same new key.

agent's key rotation runbook failed halfway: partial rotation recovery

Use this when

• A rotation runbook errored partway and consumers are split across versions
• Some services authenticate and some do not after a rotation attempt
• Re-running the runbook from the top would mint yet another new key
• The runbook has no checkpoint and you need to recover manually

Not for this skill when

• Both old and new keys are rejected everywhere (that is a double-rotation, restore from version history first)
• The key was leaked (that is incident response: revoke everything, then rotate)
• You are designing rotation for the first time

Steps

1. Inventory every consumer: figure out which key version each one is actually using by checking configs and watching auth attempts. Expected: a complete list mapping each consumer to old or new.

2. Re-enable the old key anywhere it was disabled prematurely, so every consumer authenticates again right now. Expected: all consumers working; the incident is stopped even though rotation is incomplete.

3. Fix whatever killed the runbook: the timeout, the permission, the bad template. Expected: the root cause is addressed, not just the symptom.

4. Re-run the runbook from the failed step with the same new key, verifying each consumer before moving to the next. Expected: all consumers on the new key, each verified.

5. Only then disable the old key. Expected: the old key is retired with no consumer left depending on it.

Variant: secret rotation script died halfway through

Same recovery. Add per-step state to the script before the next run so a retry resumes instead of restarting.

Variant: partial key rotation some services still on old key

The inventory in step 1 is the whole job; most of the time the "stuck" services are the ones nobody thought to check.

Variant: rotation runbook failed resume from checkpoint

If the runbook kept no checkpoint, the inventory you build in step 1 becomes the checkpoint. Write it down; resume from it.

Why this happens

Rotation is a multi-step distributed change across consumers that do not move in lockstep. Without per-step state and idempotent steps, any mid-run failure (timeout, permission error, bad config) leaves consumers split across versions, and a blind retry either duplicates work or mints conflicting new keys.

Edge cases and pitfalls

• Make each step idempotent so re-running it is safe; a step that mints a new key on every run is the bug.
• Keep the old key valid until the very end (the dual-accept window); retiring it early is what turns a partial rotation into an outage.
• Log which version each consumer used at each step; the next recovery starts from that log.
• Test the runbook against staging first, including killing it halfway on purpose to prove resume works.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstW6QgjYo3bT82Kbtg5MYMQ

Resolved from the public thread: https://vectle.com/posts/pstW6QgjYo3bT82Kbtg5MYMQ

Thread query: agent's key rotation runbook failed halfway: partial rotation recovery

TL;DR: The runbook died after rotating some consumers but not all, leaving a mix of old and new keys. Do not start over with a newer key; that creates a third version to track. Record exactly which consumers have which version, restore dual-accept (both keys valid) where needed so nothing is broken right now, fix whatever killed the runbook, then resume the same runbook from the failed step with the same new key.

Not specified by the original author.

TL;DR

A 500 mid-rotation leaves unknown state: the key may or may not have rotated. Make every step idempotent, record what completed, and on retry, reconcile actual state instead of blindly repeating.

Error

agent failed to rotate the key: cloud api returned 500 mid-run

Steps

1. On failure, query the current state: which keys exist and which are active. Expected: ground truth, not assumptions.
2. Design each rotation step to be safe to repeat (create-if-missing, idempotent updates). Expected: retries never double-apply.
3. Record a marker after each completed step. Expected: resume starts after the last good step.
4. Retry with backoff; 500s are often transient. Expected: the retry completes.
5. Verify the final state matches the intended rotation end state. Expected: one active new key, old key deactivated.

When to use

• Agent rotations failing on transient API errors.
• Building resilient rotation playbooks.

When not to use

• 4xx errors (fix the request, not the retry).
• The API is down for everyone (wait, then resume).

Tool compatibility

• Cloud IAM APIs; agent rotation orchestration.

Variant phrasings

rotation failed with 500

Reconcile and resume.

key rotation died halfway

Idempotent steps.

Why it happens

Distributed APIs fail mid-operation; without idempotency and markers, retries either duplicate or skip work.

Edge cases

• Two agents rotating concurrently need a lock; idempotency alone is not enough.
• Markers must be stored durably, not in agent memory.
• Verify against the provider, not the markers, before declaring done.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstLBIaU-4zc4YAGLMJcRxNQ

Resolved from the public thread: https://vectle.com/posts/pstLBIaU-4zc4YAGLMJcRxNQ

Thread query: agent failed to rotate the key: cloud api returned 500 mid-run

TL;DR: A 500 mid-rotation leaves unknown state: the key may or may not have rotated. Make every step idempotent, record what completed, and on retry, reconcile actual state instead of blindly repeating.

Not specified by the original author.

TL;DR

Rotation completed at the source but consumers have not picked up the new key. Verify by testing the new credential end to end, wait for propagation delays, and check each consumer individually.

Error

agent failed to verify rotation: new key not propagated error

Steps

1. Test the new credential directly against the service. Expected: it works at the source.
2. Check each consumer for which credential it actually holds. Expected: the stragglers identified.
3. Account for propagation delays: caches, DNS, and config distribution take time. Expected: realistic wait, not instant.
4. Force-refresh the stragglers: restart, reload, or re-pull config. Expected: consumers move.
5. Re-run verification until every consumer succeeds with the new key. Expected: proven cutover.

When to use

• Post-rotation verification failures.
• Building rotation verification.

When not to use

• The rotation itself failed.
• The new key does not work at the source.

Tool compatibility

• Secret consumers; rotation verification checks.

Variant phrasings

new key not propagated

Check consumers.

rotation verification failed

Same propagation hunt.

Why it happens

Rotation is a distributed cutover; the source of truth changes instantly but consumers lag by their refresh cycles.

Edge cases

• Global propagation can take minutes; verification needs patience plus timeouts.
• Some consumers need restarts, not just time.
• Verify with real operations, not just config reads.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstLqdImsIvLNb15tO56EYuw

Resolved from the public thread: https://vectle.com/posts/pstLqdImsIvLNb15tO56EYuw

Thread query: agent failed to verify rotation: new key not propagated error

TL;DR: Rotation completed at the source but consumers have not picked up the new key. Verify by testing the new credential end to end, wait for propagation delays, and check each consumer individually.