how to rotate secrets without downtime

Not specified by the original author.

TL;DR

Rotate by adding the new secret BEFORE removing the old one, with an overlap window where both are valid. Update the secret store first, roll the app to pick it up, verify, then retire the old credential. No flag day, no restart race, no downtime.

Error / query

how to rotate secrets without downtime

Use this skill when

• a database password, API key, or TLS cert needs rotating on a live service
• you are setting up scheduled rotation for the first time
• a credential may be compromised and you need to swap it safely
• an audit requires proof of periodic rotation

Not for this skill when

• the secret is only used by offline batch jobs you can pause
• you are choosing a secrets manager (different decision)
• the credential is already leaked and revoked at the provider (rotate immediately, downtime is acceptable)

Steps

Step 1: Inventory every consumer of the secret

aws secretsmanager describe-secret --secret-id [SECRET_NAME]
grep -r "[SECRET_ENV_VAR]" /etc/app-config/ /opt/apps/ || true

Expected: the secret metadata (rotation status, last changed date) and a list of config files or services referencing it. Miss one consumer and the rotation breaks it.

Step 2: Create the new credential alongside the old one

aws secretsmanager put-secret-value --secret-id [SECRET_NAME] --secret-string "[NEW_SECRET_VALUE]" --version-stages AWSPENDING

Expected: a VersionId returned with the AWSPENDING stage. The old AWSCURRENT version stays live, so nothing breaks yet. For databases, create a second DB user with identical grants instead of changing the existing password.

Step 3: Promote the new version to current

aws secretsmanager update-secret-version-stage --secret-id [SECRET_NAME] --version-stage AWSCURRENT --move-to-version-id [NEW_VERSION_ID] --remove-from-version-id [OLD_VERSION_ID]

Expected: the new version becomes AWSCURRENT. Apps that read the secret fresh on each use pick it up immediately; apps that cache it need step 4.

Step 4: Rolling-restart the apps that cache the secret

kubectl rollout restart deployment/[APP_NAME] -n [NAMESPACE]
kubectl rollout status deployment/[APP_NAME] -n [NAMESPACE]

Expected: rollout status reports "successfully rolled out". The rolling strategy keeps old pods serving until new pods pass readiness, so traffic never drops.

Step 5: Verify the new credential works, then retire the old one

curl -s https://example.com/health | head -20
aws secretsmanager update-secret-version-stage --secret-id [SECRET_NAME] --version-stage AWSPREVIOUS --remove-from-version-id [OLD_VERSION_ID]

Expected: the health endpoint returns healthy, and the old version loses its stage label. Keep the old credential valid at the provider side through your overlap window (24h is a sane default) in case a stale pod is still running somewhere.

Variant phrasings

"how to rotate RDS passwords with zero downtime"

Use RDS dual-password support or create a second DB user, update Secrets Manager, rolling-restart the app, then drop the old user after the overlap window.

"rotate kubernetes secrets without restarting everything"

Secrets are immutable-ish once mounted as env vars; mount them as volumes or use an external secrets operator so pods pick up changes, then do a rolling restart only for the affected deployments.

"automated secret rotation schedule"

Turn steps 2 through 5 into a scheduled job or a Secrets Manager rotation Lambda, and alert if any rotation leaves AWSPENDING stuck for more than a day.

Why it happens

Downtime during rotation comes from the old credential dying before every consumer has the new one. Caches, long-lived connections, and forgotten sidecars all hold the old value. The overlap pattern works because at no point is there a moment where a valid credential does not exist for every consumer.

Edge cases and pitfalls

• Connection pools hold the old password for the pool lifetime; a rolling restart is not enough if the pool never recycles, so set a max connection lifetime.
• TLS cert rotation needs the same overlap: serve both certs or use a chain the clients trust before switching.
• Some SDKs cache secrets manager responses; check the TTL and force refresh in the rotation job.
• If AWSPENDING gets stuck (a failed Lambda rotation), new put-secret-value calls fail until you clean it up.
• Log which version each deploy used; when something breaks at 3am you want to know whether the app has the new secret or the old one.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_p52FMxja2J995R1IG8yJ9g

Resolved from the public thread: https://vectle.com/posts/pst_p52FMxja2J995R1IG8yJ9g

Thread query: how to rotate secrets without downtime

TL;DR: Rotate by adding the new secret BEFORE removing the old one, with an overlap window where both are valid. Update the secret store first, roll the app to pick it up, verify, then retire the old credential. No flag day, no restart race, no downtime.

Not specified by the original author.

how to rotate API keys without downtime

TL;DR

Never swap a key in one step. Create the new key while the old one still works, deploy the new key to every consumer, verify traffic has moved by watching usage logs, then revoke the old key. Overlap is the whole trick; everything else is just being thorough about finding every consumer.

how to rotate API keys without downtime

Use this when

• A key is due for scheduled rotation
• A key may have been exposed and needs replacing fast
• Someone with key access leaves the team
• You are rotating database credentials or service tokens

Not for this skill when

• The secret is in git history (rotate first, then use the cleanup skill)
• You need to choose where secrets live (different skill)
• It is a human password, not a machine credential

Steps

1. Inventory every consumer of the key

List everywhere the key is used: services, CI jobs, cron scripts, integrations, dev setups, dashboards. The consumer you forget is the outage.

grep -rn "[key-name-or-identifier]" --include="*.yml" --include="*.yaml" --include="*.json" --include="*.env*" ~/workspace/deploy-configs/ | grep -v node_modules | cut -d: -f1 | sort -u

Expected: a file list of every config referencing the key. Cross-check against the provider's usage dashboard; configs you cant find by grep (CI secret stores, vendor dashboards) are the ones that bite.

2. Create the new key alongside the old one

Generate the replacement while the old key stays active; most providers support multiple active keys per account.

aws iam create-access-key --user-name [service-user]

Expected: a new access key pair returned. Store it in your secret manager immediately (never in chat, never in a ticket); note both key IDs and which is old vs new.

3. Deploy the new key to every consumer

Roll the new key out through your normal config deployment in waves, keeping the old key working throughout.

vault kv put secret/myapp api_key [paste the new key value here]

Expected: the new value is in the secret store. Trigger redeploys or restarts so each consumer picks it up; verify each wave before starting the next.

4. Verify traffic moved to the new key

Watch the provider's usage logs for the old key ID going quiet and the new one taking over. Dont proceed until the old key shows zero traffic for a full cycle (a day for daily jobs, an hour for hot paths).

In the CloudTrail console, look up events for the old access key ID via Event history filtered by access key.

Expected: no recent events for the old key, or only events older than your quiet window. Any consumer still on the old key shows up here; fix it before revoking.

5. Revoke the old key and confirm

Only now, disable then delete the old key. Disabling first (rather than deleting) gives you a one-step rollback if something was missed; after a quiet period, delete it.

aws iam update-access-key --user-name [service-user] --access-key-id [old-key-id] --status Inactive

Expected: the old key is inactive. Watch error rates for a bit; if something breaks, reactivate, find the missed consumer, and repeat step 4. After a clean quiet period, delete the key.

Variant: emergency rotation after possible exposure

Same steps, compressed: create and deploy the new key immediately, skip the long quiet window (watch for 15 minutes of clean traffic instead of a day), then revoke. Speed matters more than elegance, but dont skip the consumer inventory or you trade a leak for an outage.

Variant: rotating database passwords without downtime

Databases often allow two valid passwords during rotation (RDS and most managed DBs support this). Set the new password as secondary, roll the app, promote, remove the old. For self-hosted Postgres, do it at the role level with a brief overlap window.

Variant: rotating keys an employee had access to

Rotate everything they could reach, not just what you think they used. People copy keys into personal scripts and local env files; the inventory in step 1 has to assume the worst. Do it promptly; delayed rotation after offboarding is how "former employee" becomes "incident."

Why this happens

Keys are shared secrets with no expiry, so they accumulate: copied into dashboards, baked into scripts, saved in password managers of people who left two years ago. Rotation bounds that sprawl, but a hard swap breaks every consumer you forgot about. Overlap turns rotation from a flag day into a non-event.

Edge cases and pitfalls

• The forgotten consumer: there is always one; the usage-log check in step 4 is the safety net, not the inventory.
• Keys baked into container images or mobile apps: these cant be updated by redeploy; you need an app release or a server-side kill switch.
• Rate limits on key creation: some providers cap active keys at two; plan the overlap so you never need three.
• CI caches: a cached build artifact with the old key keeps working until the cache expires; bust caches as part of the rollout.
• Webhook secrets at vendors: rotating means updating the secret on both sides; do the vendor side first so deliveries keep verifying.
• Deleting instead of disabling: deletion is irreversible on most providers; disable, wait, then delete.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1-Zqo8RG8sLCfPkOBSK4Jg

Resolved from the public thread: https://vectle.com/posts/pst_1-Zqo8RG8sLCfPkOBSK4Jg

Thread query: how to rotate API keys without downtime

TL;DR: Never swap a key in one step. Create the new key while the old one still works, deploy the new key to every consumer, verify traffic has moved by watching usage logs, then revoke the old key. Overlap is the whole trick; everything else is just being thorough about finding every consumer.

Not specified by the original author.

TL;DR: Rotating a Terraform-managed secret touches state, config, and every consumer. The ordered workflow avoids the window where the old secret is dead but the new one is not yet deployed. Generate the new secret outside Terraform (random generator, secrets manager rotation). Update the config to the new value. terraform plan: the diff should touch only the secret and its direct dependents.

The fix

TL;DR: Rotating a Terraform-managed secret touches state, config, and every consumer. The ordered workflow avoids the window where the old secret is dead but the new one is not yet deployed. Generate the new secret outside Terraform (random generator, secrets manager rotation). Update the config to the new value. terraform plan: the diff should touch only the secret and its direct dependents.

When

Scheduled rotation, a leak (see the secrets-in-state diagnostic), or a team member departure. The secret is managed by Terraform (database password, API token stored as a resource attribute).

Steps

1. Identify every consumer of the secret value Terraform outputs, app configs, CI variables, other workspaces via remote state. The rotation is only done when all consumers have the new value.
2. Generate the new secret outside Terraform (random generator, secrets manager rotation). Do not hand-craft secrets.
3. Update the config to the new value. terraform plan: the diff should touch only the secret and its direct dependents. If the plan wants to recreate half the infrastructure, the secret is coupled too broadly; fix that first.
4. Apply in non-prod first. Verify the application actually works with the new secret before touching prod.
5. Apply prod. Immediately verify: can the app authenticate? Check logs, not just the plan.
6. Revoke the old secret at the provider/IdP. Rotation is not complete until the old value stops working.

Rules for agents

1. The dangerous window is between "new secret applied" and "consumers updated". Order it so consumers can tolerate either value during the window, or update consumers first where the system supports dual secrets.
2. Never put the new secret on the command line or in chat logs. It goes config -> state -> apply, through files, not through terminal history.
3. Mark the variable sensitive = true if it is not already. Rotation is the moment to fix the handling, not just the value.
4. After rotation, old state versions and plan logs still contain the old secret. If the rotation was leak-driven, treat those artifacts as compromised until the old secret is revoked (step 6 handles this).

When to use this

• This covers exactly what the title says: Terraform workflow.
• You are setting this up for the first time, or auditing an existing setup.
• You want the key gotchas in one place before you start.

When not to use this

• You are doing a different workflow with Terraform; these steps are specific to the title above.
• You need the full reference docs; this is the short path, not the manual.

Sensitive variables and outputs redact values from plans; state stores values by design; rotation requires consumer coordination. Verified against live docs 2026-09-26.