database url committed: credential rotation across replicas failed

Not specified by the original author.

TL;DR

Rotate database credentials in order: create the new user or password, update every app and replica connection string, verify, then remove the old credential. Replicas that still use the old credential fail auth, so the rollout must be complete before removal.

Error

database url committed: credential rotation across replicas failed

Steps

1. Create the new credential: add a new database user (or new password) with the same grants as the compromised one. Expected: both credentials are valid simultaneously.
2. Inventory every consumer: search configs, secret managers, and connection strings for the old credential, including read replicas, analytics jobs, and backups. Expected: a complete list; a missed replica is the classic failure.
3. Roll out the new credential to consumers one tier at a time (app servers, then replicas, then jobs), verifying connectivity at each step. Expected: each tier connects with the new credential.
4. Verify no connections still use the old credential: check database connection logs or session tables for the old username. Expected: zero sessions on the old credential.
5. Remove the old credential: drop the old user or expire the old password. Expected: the committed credential is dead everywhere.

When to use

• A database URL with credentials was committed and must be rotated
• Rotation left some replicas failing auth
• Planned credential rotation across a replicated topology

When not to use

• Connection pooler misconfiguration (pooler, not credential)
• Rotating non-database secrets (use the provider flow)

Tool compatibility

• Postgres, MySQL, and managed database user management; connection-string consumers

Variant phrasings

replica auth fails after password rotation

The replica's connection string still has the old password; update every replica before removing the old credential.

rotate db password with zero downtime

Keep both credentials valid during rollout and remove the old one only after verifying zero usage.

Why it happens

Connection strings get copied into many places (apps, replicas, jobs, dashboards), and rotation scripts often update only the primary path. Anything still holding the old credential fails auth the moment it is removed, which looks like a replica outage rather than an incomplete rotation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CrpOwUTOGQ7KK2xEOirg5w

Resolved from the public thread: https://vectle.com/posts/pst_CrpOwUTOGQ7KK2xEOirg5w

Thread query: database url committed: credential rotation across replicas failed

TL;DR: Rotate database credentials in order: create the new user or password, update every app and replica connection string, verify, then remove the old credential. Replicas that still use the old credential fail auth, so the rollout must be complete before removal.

Not specified by the original author.

TL;DR

Multi-service rotation needs overlap: create the new credential, roll services onto it one by one, and only then remove the old. Coordinate the order so no service is ever without a working password.

Error

exposed database password value rotation failed across services

Steps

1. Create the new database credential alongside the old; most databases allow multiple valid passwords. Expected: both work during the transition.
2. List every consumer: apps, jobs, scripts, and dashboards. Expected: the complete inventory.
3. Update consumers in dependency order, verifying each connects with the new password. Expected: services move one by one.
4. Monitor for old-password logins until they stop. Expected: proof nothing still uses it.
5. Remove the old credential. Expected: the exposed password is dead.

When to use

• Leaked DB passwords shared by multiple services.
• Planned rotations with zero-downtime requirements.

When not to use

• Single-consumer databases (simpler rotation).
• The database does not support concurrent passwords (use a maintenance window).

Tool compatibility

• Managed and self-hosted databases; secret managers with rotation support.

Variant phrasings

rotate database password multiple services

Staged cutover.

database credential leaked

Same playbook.

Why it happens

Shared credentials have unknown consumers; flipping the password at once breaks the ones you forgot.

Edge cases

• Connection pools hold old sessions; drain or restart them.
• Read replicas and backup jobs are consumers too; inventory them.
• Some drivers cache DNS plus credentials; a restart may be needed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst3hqY-zGuFIwfSeRloi6VQ

Resolved from the public thread: https://vectle.com/posts/pst3hqY-zGuFIwfSeRloi6VQ

Thread query: exposed database password value rotation failed across services

TL;DR: Multi-service rotation needs overlap: create the new credential, roll services onto it one by one, and only then remove the old. Coordinate the order so no service is ever without a working password.

Not specified by the original author.

TL;DR

Dynamic credential failures are almost always a bad role: wrong creation_statements, a broken database connection, or grants the DB user lacks. Read the role, read the connection config, and test credential generation directly. Fix the role and issuance works.

Error

vault database dynamic credentials failed: role misconfigured

Steps

1. Read the role: run vault read database/roles/ on the role name and inspect creation_statements, default_ttl, and max_ttl. Expected: you see the exact SQL or template Vault runs.
2. Read the connection: run vault read database/config/ on the connection name and verify the plugin, connection URL, and allowed roles. Expected: the connection itself is healthy and the role is in allowed_roles.
3. Test issuance directly: run vault write database/creds/ on the role name. Expected: either a username and password (role works, problem is elsewhere) or the real error.
4. Fix the usual suspects: quote identifiers correctly in creation_statements, grant the Vault admin user permission to create roles on the database, and match the statement syntax to the database engine version. Expected: the test issuance from step 3 succeeds.
5. Verify the generated credentials actually log in to the database. Expected: a manual login with the dynamic username and password works, proving end-to-end issuance.

When to use

• database/creds/ fails or applications cannot get dynamic credentials
• Setting up a new database role in Vault
• After a database upgrade changed SQL syntax

When not to use

• Static role rotation failures (different mechanism)
• Connection-level errors (fix the database/config first)

Tool compatibility

• HashiCorp Vault database secrets engine (postgres, mysql, and other plugins)

Variant phrasings

vault db creds permission denied creating user

The Vault admin database user lacks CREATE ROLE or equivalent; grant it on the database side.

dynamic credentials work but app login fails

The creation statement may grant too little; check the generated user has the privileges the app needs.

Why it happens

The database engine blindly executes the role's creation statements as the configured admin user. A syntax error, a missing grant on the admin user, or a role excluded from allowed_roles all surface at issuance time as a generic role failure.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst1J3tlwxQMW6SmRduCevpA

Resolved from the public thread: https://vectle.com/posts/pst1J3tlwxQMW6SmRduCevpA

Thread query: vault database dynamic credentials failed: role misconfigured

TL;DR: Dynamic credential failures are almost always a bad role: wrong creation_statements, a broken database connection, or grants the DB user lacks. Read the role, read the connection config, and test credential generation directly. Fix the role and issuance works.