agent double-rotated a secret value old and new keys both rejected

agent double-rotated a secret value old and new keys both rejected

TL;DR

The agent created a second new secret before the first new one finished rolling out, and disabled the old one too early, so no valid credential remains anywhere. Stop the rotation jobs first. Restore the last known-good version from the secret store's version history, confirm every consumer authenticates with it, then run one clean rotation with a dual-accept window where old and new are both valid until every consumer has moved.

agent double-rotated a secret value old and new keys both rejected

Use this when

• Two rotations ran close together and now nothing authenticates
• A rotation script ran twice (a retry plus a cron, or two agents racing)
• Consumers are split: some reject the old value, some reject the new one
• The secret store shows several recent versions and nobody knows which is live

Not for this skill when

• Only the new value fails and the old one still works (that is a bad new value, just roll back)
• The secret was leaked and must be revoked immediately (that is incident response, not rotation recovery)
• You are setting up rotation for the first time

Steps

1. Freeze all rotation jobs and automation touching this secret. A third rotation right now makes recovery harder. Expected: no new versions appear while you work.

2. List the secret's versions in the store and note which version each consumer actually has. AWS Secrets Manager lists version IDs with their stage labels; Vault KV metadata lists versions the same way. Expected: a short list of recent versions with created times, and a map of consumer to version.

3. Restore the last known-good version as the current one, using the store's version-history feature rather than typing the value by hand. Expected: reads from the store return the working value again.

4. Verify every consumer authenticates with the restored value before touching anything else. Expected: successful auth from each consumer, no errors in their logs.

5. Run one clean rotation: create the new version, update consumers one by one and verify each, keep the old value valid until all consumers confirm, then deprecate the old version. Expected: all consumers on the new value, old version marked deprecated, nothing broken.

Variant: secret rotation broke production both keys invalid

Same recovery. Resist the urge to mint a third key; restore from version history first, because the known-good value is already proven.

Variant: AWS Secrets Manager rotation failed after second rotation

Use the version stages: find the version carrying the previous stage label, promote it back to current, and disable the rotation trigger until recovery is done.

Variant: Vault KV old secret version no longer works

Read the KV metadata to list versions, then write the known-good version's data back as a new version. Fix the automation that deleted or overwrote versions before re-enabling it.

Why this happens

Safe rotation has three phases: create the new value, propagate it to every consumer, then retire the old one. A second rotation during the propagate phase collapses the sequence, and retiring the old value early removes the only fallback. Two agents or a retry loop can trigger the second rotation without anyone noticing.

Edge cases and pitfalls

• Some stores auto-expire old versions after N days; check the retention setting during recovery or the fallback may already be gone.
• Consumers cache credentials in memory; restoring the store value is not enough until each consumer reloads or restarts.
• Audit what triggered the second rotation (overlapping crons, a retry plus the original, two agents) and fix the trigger, or it will happen again.
• Keep the dual-accept window explicit and time-boxed; both-valid-forever becomes a forgotten second credential.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_PMniJmkUxo03oftBtajPlQ