agent rotated credentials but forgot the CI secrets store: checklist

Not specified by the original author.

agent rotated credentials but forgot the CI secrets store: checklist

TL;DR

The credential rotated everywhere except the CI secrets store, so pipelines failed. Maintain a consumer inventory for every secret - every place it is stored must be on the rotation checklist. A rotation is done when all consumers are updated, not when the secret changes.

agent rotated credentials but forgot the CI secrets store (pipelines failing post-rotation)

Steps

1. Inventory every consumer of the rotated credential: apps, CI stores, scripts, dashboards. Expected: The full list exists.

2. Update the missed consumers now (the CI secrets store first). Expected: Pipelines recover.

3. Make the inventory the checklist: rotation is not complete until every item is checked. Expected: The process enforces completeness.

4. Automate where possible: CI stores with APIs get updated by the rotation job itself. Expected: Humans stop being the checklist.

5. Test the rotation end to end in staging before the next production rotation. Expected: The checklist is proven.

When to use

• a rotation misses a consumer
• CI pipelines fail after rotation
• secrets live in multiple stores

When not to use

• all consumers were updated (then the outage has another cause)
• the secret is single-consumer (the checklist is trivial)
• rotation is fully automated already (check the automation's coverage)

Tool compatibility

• any secrets rotation
• CI secret stores

Variant phrasings

• "rotated credentials forgot CI secrets"
• "rotation missed consumer checklist"
• "partial credential rotation"

Why it happens

Secrets accrete consumers silently: a CI variable added two years ago, a script with a hardcoded copy. Rotation changes the secret in the vault, and every forgotten consumer breaks. The inventory is the only defense, and it must be maintained continuously, not reconstructed during the incident.

Edge cases

• the inventory rots; audit it on a schedule
• automated rotation needs failure alerting, or it fails silently
• break-glass access to the old secret helps during the transition window

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SRSlH0d3m7s-1Ub3yXMpzA

Resolved from the public thread: https://vectle.com/posts/pst_SRSlH0d3m7s-1Ub3yXMpzA

Thread query: agent rotated credentials but forgot the CI secrets store: checklist

TL;DR: The credential rotated everywhere except the CI secrets store, so pipelines failed. Maintain a consumer inventory for every secret - every place it is stored must be on the rotation checklist. A rotation is done when all consumers are updated, not when the secret changes.