secret rotation failed: app still reading old value
Fixes secret rotation where the app still reads the old value: find the caching layer and force a reload. Use when rotation completed but apps serve stale credentials. Not for failed rotations.
TL;DR
The new credential is live but the app cached the old one at startup or in memory. Identify where the value is cached, trigger a reload or rolling restart, and verify the live value changed.
Error
secret rotation failed: app still reading old valueSteps
- Confirm the new value is live at the source of truth. Expected: the manager returns the new credential.
- Check what the app actually holds: a debug endpoint or log (redacted) showing which credential is in use. Expected: the old value confirmed in the app.
- Find the cache: process env at startup, in-memory client, connection pool, or sidecar. Expected: the layer identified.
- Reload or rolling-restart to pick up the new value. Expected: the app reads fresh.
- Verify with a real operation using the new credential. Expected: success proves the cutover.
When to use
- Post-rotation staleness in apps.
- Designing rotation-safe consumers.
When not to use
- The rotation itself failed (fix the rotation).
- The new value is wrong (fix the value).
Tool compatibility
- Any app consuming rotated secrets; secret managers with versioning.
Variant phrasings
app using old secret after rotation
Cache or restart.
rotation didn't propagate to app
Same reload fix.
Why it happens
Apps read secrets once and cache them; rotation changes the source, not the cache. Without a reload signal, the app never notices.
Edge cases
- Connection pools hold authenticated sessions; they need draining, not just config reload.
- Multiple replicas restart at different times; roll them together.
- Some SDKs cache aggressively; check the client's refresh behavior.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_SI65PvulzdLZZ7bXk53gRQ