exposed database password value rotation failed across services
Rotates an exposed database password across services without downtime: staged cutover with dual credentials. Use when a DB password leaked and many services use it. Not for single-service rotations.
TL;DR
Multi-service rotation needs overlap: create the new credential, roll services onto it one by one, and only then remove the old. Coordinate the order so no service is ever without a working password.
Error
exposed database password value rotation failed across servicesSteps
- Create the new database credential alongside the old; most databases allow multiple valid passwords. Expected: both work during the transition.
- List every consumer: apps, jobs, scripts, and dashboards. Expected: the complete inventory.
- Update consumers in dependency order, verifying each connects with the new password. Expected: services move one by one.
- Monitor for old-password logins until they stop. Expected: proof nothing still uses it.
- Remove the old credential. Expected: the exposed password is dead.
When to use
- Leaked DB passwords shared by multiple services.
- Planned rotations with zero-downtime requirements.
When not to use
- Single-consumer databases (simpler rotation).
- The database does not support concurrent passwords (use a maintenance window).
Tool compatibility
- Managed and self-hosted databases; secret managers with rotation support.
Variant phrasings
rotate database password multiple services
Staged cutover.
database credential leaked
Same playbook.
Why it happens
Shared credentials have unknown consumers; flipping the password at once breaks the ones you forgot.
Edge cases
- Connection pools hold old sessions; drain or restart them.
- Read replicas and backup jobs are consumers too; inventory them.
- Some drivers cache DNS plus credentials; a restart may be needed.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst3hqY-zGuFIwfSeRloi6VQ