Architecture:
1. **Lifetime policy on the secret.** Set expiry (e.g. 90 days) and a rotation trigger: Key Vault supports auto-rotation for certain secret types (storage account keys, via the managed storage account feature). For generic secrets, use the near-expiry event:
- Event Grid system topic on the vault, subscribing to `Microsoft.KeyVault.SecretNearExpiry`.
2. **Rotation function.** An Azure Function (Event Grid trigger) that:
- Generates/obtains the new credential (call the downstream service's API to create a new key/password).
- Writes it as a new secret version: `az keyvault secret set` or SDK.
- Updates the consumer if the consumer cannot read "latest" (ideally consumers always read latest enabled version, so this step is a no-op).
- Disables (not deletes) the old version after a grace period.
3. **Consumer discipline.** Every consumer reads the secret at startup AND refreshes periodically (or on auth failure), always the latest enabled version. A consumer that caches forever defeats rotation.
4. **Alerting.** Alert on `SecretExpired` events and on rotation-function failures. A silent rotation failure becomes an outage at expiry.
Traps:
- Rotating the secret the rotation function itself uses to authenticate = lockout. The function's identity must be a managed identity, never a vault-stored secret.
- Two writers racing (manual set + auto rotation) create version confusion. One writer per secret.
- Test rotation in a staging vault first; a bug that writes garbage as the newest version breaks every consumer at once.
Verify: force a near-expiry event (or temporarily short expiry), watch the function run, confirm consumers pick up the new version with no restart.