how to rotate Kubernetes service account tokens safely

TL;DR

Safe token rotation is a two-phase dance: create the new credential alongside the old one, roll workloads onto it, then remove the old one. Never delete the old token before the new one is serving traffic. For modern clusters, prefer time-bound projected tokens so rotation happens automatically.

The query

how to rotate Kubernetes service account tokens safely

Use this when

• A service account token may be leaked or compromised
• Periodic credential rotation is required by policy
• Migrating from legacy long-lived tokens to bound tokens
• Auditing which workloads use which tokens

Not for when

• Rotating user certificates in kubeconfig files
• Cloud IAM credential rotation (different system)
• One-off token creation for debugging

Steps

Step 1: Inventory where the token is used

Find every workload, CI job, and external system using the service account token. Check mounted secrets, CI variables, and external integrations. Rotating a token you did not know was shared is how outages happen. Expected output: a complete list of consumers of the token.

Step 2: Issue the new token alongside the old

Create the new token without deleting the old one. Both credentials are valid during the transition; nothing breaks yet. For projected/bound tokens, update the pod spec to request the new token volume. Expected output: new credential issued and distributed; old credential still working.

Step 3: Migrate consumers one by one

Update each consumer to the new token, starting with the least critical. Verify each one works after the switch before moving to the next. For in-cluster workloads this means rolling the pods; for external systems it means updating their stored credential. Expected output: all consumers running on the new token, confirmed working.

Step 4: Monitor for stragglers using the old token

Watch API server audit logs for authentications with the old token. Any remaining usage names a consumer you missed in step 1. Do not proceed until usage drops to zero. Expected output: zero authentications with the old token over a full usage cycle.

Step 5: Revoke the old token and document

Delete the old token secret or credential. Record the rotation (date, reason, who did it) for the audit trail. If policy requires periodic rotation, schedule the next one now. Expected output: old token revoked; rotation logged; next rotation scheduled.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XfE2KhzkN8ElgH0ZGnJl0A