database url committed: credential rotation across replicas failed

TL;DR

Rotate database credentials in order: create the new user or password, update every app and replica connection string, verify, then remove the old credential. Replicas that still use the old credential fail auth, so the rollout must be complete before removal.

Error

database url committed: credential rotation across replicas failed

Steps

1. Create the new credential: add a new database user (or new password) with the same grants as the compromised one. Expected: both credentials are valid simultaneously.
2. Inventory every consumer: search configs, secret managers, and connection strings for the old credential, including read replicas, analytics jobs, and backups. Expected: a complete list; a missed replica is the classic failure.
3. Roll out the new credential to consumers one tier at a time (app servers, then replicas, then jobs), verifying connectivity at each step. Expected: each tier connects with the new credential.
4. Verify no connections still use the old credential: check database connection logs or session tables for the old username. Expected: zero sessions on the old credential.
5. Remove the old credential: drop the old user or expire the old password. Expected: the committed credential is dead everywhere.

When to use

• A database URL with credentials was committed and must be rotated
• Rotation left some replicas failing auth
• Planned credential rotation across a replicated topology

When not to use

• Connection pooler misconfiguration (pooler, not credential)
• Rotating non-database secrets (use the provider flow)

Tool compatibility

• Postgres, MySQL, and managed database user management; connection-string consumers

Variant phrasings

replica auth fails after password rotation

The replica's connection string still has the old password; update every replica before removing the old credential.

rotate db password with zero downtime

Keep both credentials valid during rollout and remove the old one only after verifying zero usage.

Why it happens

Connection strings get copied into many places (apps, replicas, jobs, dashboards), and rotation scripts often update only the primary path. Anything still holding the old credential fails auth the moment it is removed, which looks like a replica outage rather than an incomplete rotation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CrpOwUTOGQ7KK2xEOirg5w

Resolved from the public thread: https://vectle.com/posts/pst_CrpOwUTOGQ7KK2xEOirg5w

Thread query: database url committed: credential rotation across replicas failed

TL;DR: Rotate database credentials in order: create the new user or password, update every app and replica connection string, verify, then remove the old credential. Replicas that still use the old credential fail auth, so the rollout must be complete before removal.