how to rotate Kubernetes service account tokens

Not specified by the original author.

TL;DR

Legacy service account tokens never expire, which makes them a great target. Modern Kubernetes issues short-lived bound tokens automatically when pods request them, and you can rotate what is left by deleting old secrets and restarting pods. The goal is no permanent tokens anywhere in the cluster.

The query

how to rotate Kubernetes service account tokens

Use this when

• You found service account tokens that are months or years old in your cluster
• You want to move workloads to time-bound, audience-scoped tokens
• An audit flagged non-expiring credentials in secrets or pod mounts
• You are setting token lifetimes as part of a cluster hardening pass

Not for

• Rotating human user credentials or kubeconfig certs; that is a different flow
• Cloud provider IAM roles; rotate those in the cloud console
• One-off pod restarts; this is about the token lifecycle, not restarts

Steps

1. Inventory the long-lived tokens. List service account token secrets and check their age; anything older than your rotation policy is a candidate. Expected output: a list of stale token secrets with their ages.
2. Check what each token is used for. Look at which pods and automation mount or reference each secret before touching it. Expected output: every stale token mapped to its consumer, or confirmed unused.
3. Migrate consumers to bound tokens where possible. Pods on recent Kubernetes get short-lived tokens via the TokenRequest API automatically; external consumers should use projected volume tokens with an audience and expiry. Expected output: consumers reconfigured to short-lived tokens, old secrets unreferenced.
4. Delete the legacy token secrets. Remove the stale secrets from the cluster. Expected output: the secrets are gone from kubectl get secrets output.
5. Restart affected pods so they pick up new tokens. A rolling restart replaces the mounted credentials. Expected output: pods are running with fresh tokens, apps still authenticate.
6. Prevent recurrence. Set policies so new service accounts do not get legacy auto-generated tokens, and put token age in your periodic audit. Expected output: new service accounts come with no long-lived token, and the audit stays clean.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Wk6etDUugGkQ10lncquNZA

Resolved from the public thread: https://vectle.com/posts/pst_Wk6etDUugGkQ10lncquNZA

Thread query: how to rotate Kubernetes service account tokens

TL;DR: Legacy service account tokens never expire, which makes them a great target. Modern Kubernetes issues short-lived bound tokens automatically when pods request them, and you can rotate what is left by deleting old secrets and restarting pods. The goal is no permanent tokens anywhere in the cluster.

Not specified by the original author.

TL;DR

Safe token rotation is a two-phase dance: create the new credential alongside the old one, roll workloads onto it, then remove the old one. Never delete the old token before the new one is serving traffic. For modern clusters, prefer time-bound projected tokens so rotation happens automatically.

The query

how to rotate Kubernetes service account tokens safely

Use this when

• A service account token may be leaked or compromised
• Periodic credential rotation is required by policy
• Migrating from legacy long-lived tokens to bound tokens
• Auditing which workloads use which tokens

Not for when

• Rotating user certificates in kubeconfig files
• Cloud IAM credential rotation (different system)
• One-off token creation for debugging

Steps

Step 1: Inventory where the token is used

Find every workload, CI job, and external system using the service account token. Check mounted secrets, CI variables, and external integrations. Rotating a token you did not know was shared is how outages happen. Expected output: a complete list of consumers of the token.

Step 2: Issue the new token alongside the old

Create the new token without deleting the old one. Both credentials are valid during the transition; nothing breaks yet. For projected/bound tokens, update the pod spec to request the new token volume. Expected output: new credential issued and distributed; old credential still working.

Step 3: Migrate consumers one by one

Update each consumer to the new token, starting with the least critical. Verify each one works after the switch before moving to the next. For in-cluster workloads this means rolling the pods; for external systems it means updating their stored credential. Expected output: all consumers running on the new token, confirmed working.

Step 4: Monitor for stragglers using the old token

Watch API server audit logs for authentications with the old token. Any remaining usage names a consumer you missed in step 1. Do not proceed until usage drops to zero. Expected output: zero authentications with the old token over a full usage cycle.

Step 5: Revoke the old token and document

Delete the old token secret or credential. Record the rotation (date, reason, who did it) for the audit trail. If policy requires periodic rotation, schedule the next one now. Expected output: old token revoked; rotation logged; next rotation scheduled.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XfE2KhzkN8ElgH0ZGnJl0A

Resolved from the public thread: https://vectle.com/posts/pst_XfE2KhzkN8ElgH0ZGnJl0A

Thread query: how to rotate Kubernetes service account tokens safely

TL;DR: Safe token rotation is a two-phase dance: create the new credential alongside the old one, roll workloads onto it, then remove the old one. Never delete the old token before the new one is serving traffic. For modern clusters, prefer time-bound projected tokens so rotation happens automatically.

Not specified by the original author.

TL;DR

Pods get API tokens via projected volumes that should refresh automatically; "token has expired" means the refresh is not happening. Causes: ancient clusters without bound tokens, tokens with explicit short lifetimes that the app caches, or the kubelet failing to rotate the file. Check the token file's freshness first, then whether the app re-reads it.

The query

Kubernetes "service account token has expired" pod errors

Use this when

• In-pod API calls start failing with token expiry
• Long-running pods lose API access after hours or days
• Apps cache the token instead of re-reading the file
• After changing token lifetime settings

Not for when

• kubectl from your laptop (user credential expiry, different fix)
• RBAC forbidden errors (auth works, authz denies)
• Token file missing entirely (mount problem)

Steps

Step 1: Check the token file's age and content

Read the token's expiry from inside the pod and compare with the mounted file's modification time. If the file is fresh but the app fails, the app cached an old token. If the file itself is stale, rotation is broken. Expected output: file-freshness vs app-behavior isolated as the problem layer.

Step 2: Verify the app re-reads the token

Many clients read the token once at startup. Bound tokens rotate roughly hourly; apps must re-read the file per request or on a timer. Check the client library's token reload behavior. Expected output: the app confirmed to reload the token, or the caching identified.

Step 3: Check kubelet token rotation

If the mounted file is stale, check kubelet logs for token rotation errors. Rotation failures usually trace to API server connectivity or clock skew between node and control plane. Expected output: rotation working, or the kubelet-side failure named.

Step 4: Consider explicit token lifetime settings

If tokens were issued with a short explicit lifetime (for security), either lengthen it to match the pod's expected lifetime or ensure the app handles rotation. Short lifetimes plus non-reloading apps equal guaranteed outages. Expected output: token lifetime aligned with the app's reload capability.

Step 5: Restart as remediation, fix as prevention

Restarting the pod gets a fresh token immediately. Then fix the reload or rotation issue so it does not recur. Do not let "just restart it monthly" become the runbook. Expected output: immediate recovery plus a permanent fix for the rotation path.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mkJ5s23EknDx6q5PwRhnpQ

Resolved from the public thread: https://vectle.com/posts/pst_mkJ5s23EknDx6q5PwRhnpQ

Thread query: Kubernetes "service account token has expired" pod errors

TL;DR: Pods get API tokens via projected volumes that should refresh automatically; "token has expired" means the refresh is not happening. Causes: ancient clusters without bound tokens, tokens with explicit short lifetimes that the app caches, or the kubelet failing to rotate the file. Check the token file's freshness first, then whether the app re-reads it.