# agent's secret rotation broke staging first: canary rotation strategy

## TL;DR

The rotation went to all environments at once and broke staging, which was actually the lucky outcome: it proved the rotation was bad before production. Formalize it: rotate in staging first, verify, then promote to production. Staging exists to break first.

```text
agent's secret rotation broke staging first (rotation flawed)
```

## Steps

1. Fix staging with the corrected rotation procedure.
   Expected: Staging recovers.

2. Write down what was wrong with the rotation: missed consumer, bad format, wrong scope.
   Expected: The flaw is named.

3. Make canary rotation the policy: staging first, verify fully, then production.
   Expected: The sequence is enforced.

4. Define the staging verification checklist: all consumers working, no errors.
   Expected: The gate has teeth.

5. Run the next production rotation through the canary.
   Expected: Production is protected by the canary.

## When to use

- a rotation breaks an environment
- rotations go everywhere at once
- staging exists but is not used for rotations

## When not to use

- staging and production differ too much for the canary to be meaningful (fix that first)
- the rotation is fully automated and tested (the canary is built in)
- there is no staging (then canary in a staging-like slice)

## Tool compatibility

- any secret rotation
- multi-environment setups

## Variant phrasings

- "secret rotation broke staging"
- "canary rotation strategy"
- "rotation failed staging first"

## Why it happens

Rotations touch every consumer of a secret, and the consumer list is always incomplete. Breaking staging first is the system working as designed: the flaw surfaces where it is cheap. Rotating everywhere at once turns every flaw into a production incident. The canary is not extra caution; it is the mechanism that makes rotation safe.

## Edge cases

- staging must mirror production's consumer set, or the canary proves nothing
- the verification checklist needs to be real; a quick glance is not verification
- document the canary results; they are the production go/no-go

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0_vrpdSQ1A5YCvCIpq4t4A