how to rotate secrets without downtime

TL;DR

Rotate by adding the new secret BEFORE removing the old one, with an overlap window where both are valid. Update the secret store first, roll the app to pick it up, verify, then retire the old credential. No flag day, no restart race, no downtime.

Error / query

how to rotate secrets without downtime

Use this skill when

• a database password, API key, or TLS cert needs rotating on a live service
• you are setting up scheduled rotation for the first time
• a credential may be compromised and you need to swap it safely
• an audit requires proof of periodic rotation

Not for this skill when

• the secret is only used by offline batch jobs you can pause
• you are choosing a secrets manager (different decision)
• the credential is already leaked and revoked at the provider (rotate immediately, downtime is acceptable)

Steps

Step 1: Inventory every consumer of the secret

aws secretsmanager describe-secret --secret-id [SECRET_NAME]
grep -r "[SECRET_ENV_VAR]" /etc/app-config/ /opt/apps/ || true

Expected: the secret metadata (rotation status, last changed date) and a list of config files or services referencing it. Miss one consumer and the rotation breaks it.

Step 2: Create the new credential alongside the old one

aws secretsmanager put-secret-value --secret-id [SECRET_NAME] --secret-string "[NEW_SECRET_VALUE]" --version-stages AWSPENDING

Expected: a VersionId returned with the AWSPENDING stage. The old AWSCURRENT version stays live, so nothing breaks yet. For databases, create a second DB user with identical grants instead of changing the existing password.

Step 3: Promote the new version to current

aws secretsmanager update-secret-version-stage --secret-id [SECRET_NAME] --version-stage AWSCURRENT --move-to-version-id [NEW_VERSION_ID] --remove-from-version-id [OLD_VERSION_ID]

Expected: the new version becomes AWSCURRENT. Apps that read the secret fresh on each use pick it up immediately; apps that cache it need step 4.

Step 4: Rolling-restart the apps that cache the secret

kubectl rollout restart deployment/[APP_NAME] -n [NAMESPACE]
kubectl rollout status deployment/[APP_NAME] -n [NAMESPACE]

Expected: rollout status reports "successfully rolled out". The rolling strategy keeps old pods serving until new pods pass readiness, so traffic never drops.

Step 5: Verify the new credential works, then retire the old one

curl -s https://example.com/health | head -20
aws secretsmanager update-secret-version-stage --secret-id [SECRET_NAME] --version-stage AWSPREVIOUS --remove-from-version-id [OLD_VERSION_ID]

Expected: the health endpoint returns healthy, and the old version loses its stage label. Keep the old credential valid at the provider side through your overlap window (24h is a sane default) in case a stale pod is still running somewhere.

Variant phrasings

"how to rotate RDS passwords with zero downtime"

Use RDS dual-password support or create a second DB user, update Secrets Manager, rolling-restart the app, then drop the old user after the overlap window.

"rotate kubernetes secrets without restarting everything"

Secrets are immutable-ish once mounted as env vars; mount them as volumes or use an external secrets operator so pods pick up changes, then do a rolling restart only for the affected deployments.

"automated secret rotation schedule"

Turn steps 2 through 5 into a scheduled job or a Secrets Manager rotation Lambda, and alert if any rotation leaves AWSPENDING stuck for more than a day.

Why it happens

Downtime during rotation comes from the old credential dying before every consumer has the new one. Caches, long-lived connections, and forgotten sidecars all hold the old value. The overlap pattern works because at no point is there a moment where a valid credential does not exist for every consumer.

Edge cases and pitfalls

• Connection pools hold the old password for the pool lifetime; a rolling restart is not enough if the pool never recycles, so set a max connection lifetime.
• TLS cert rotation needs the same overlap: serve both certs or use a chain the clients trust before switching.
• Some SDKs cache secrets manager responses; check the TTL and force refresh in the rotation job.
• If AWSPENDING gets stuck (a failed Lambda rotation), new put-secret-value calls fail until you clean it up.
• Log which version each deploy used; when something breaks at 3am you want to know whether the app has the new secret or the old one.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_p52FMxja2J995R1IG8yJ9g