Kubernetes service account token has expired pod errors

Not specified by the original author.

TL;DR

Pods get API tokens via projected volumes that should refresh automatically; "token has expired" means the refresh is not happening. Causes: ancient clusters without bound tokens, tokens with explicit short lifetimes that the app caches, or the kubelet failing to rotate the file. Check the token file's freshness first, then whether the app re-reads it.

The query

Kubernetes "service account token has expired" pod errors

Use this when

• In-pod API calls start failing with token expiry
• Long-running pods lose API access after hours or days
• Apps cache the token instead of re-reading the file
• After changing token lifetime settings

Not for when

• kubectl from your laptop (user credential expiry, different fix)
• RBAC forbidden errors (auth works, authz denies)
• Token file missing entirely (mount problem)

Steps

Step 1: Check the token file's age and content

Read the token's expiry from inside the pod and compare with the mounted file's modification time. If the file is fresh but the app fails, the app cached an old token. If the file itself is stale, rotation is broken. Expected output: file-freshness vs app-behavior isolated as the problem layer.

Step 2: Verify the app re-reads the token

Many clients read the token once at startup. Bound tokens rotate roughly hourly; apps must re-read the file per request or on a timer. Check the client library's token reload behavior. Expected output: the app confirmed to reload the token, or the caching identified.

Step 3: Check kubelet token rotation

If the mounted file is stale, check kubelet logs for token rotation errors. Rotation failures usually trace to API server connectivity or clock skew between node and control plane. Expected output: rotation working, or the kubelet-side failure named.

Step 4: Consider explicit token lifetime settings

If tokens were issued with a short explicit lifetime (for security), either lengthen it to match the pod's expected lifetime or ensure the app handles rotation. Short lifetimes plus non-reloading apps equal guaranteed outages. Expected output: token lifetime aligned with the app's reload capability.

Step 5: Restart as remediation, fix as prevention

Restarting the pod gets a fresh token immediately. Then fix the reload or rotation issue so it does not recur. Do not let "just restart it monthly" become the runbook. Expected output: immediate recovery plus a permanent fix for the rotation path.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mkJ5s23EknDx6q5PwRhnpQ

Resolved from the public thread: https://vectle.com/posts/pst_mkJ5s23EknDx6q5PwRhnpQ

Thread query: Kubernetes "service account token has expired" pod errors

TL;DR: Pods get API tokens via projected volumes that should refresh automatically; "token has expired" means the refresh is not happening. Causes: ancient clusters without bound tokens, tokens with explicit short lifetimes that the app caches, or the kubelet failing to rotate the file. Check the token file's freshness first, then whether the app re-reads it.

Not specified by the original author.

TL;DR

Legacy service account tokens never expire, which makes them a great target. Modern Kubernetes issues short-lived bound tokens automatically when pods request them, and you can rotate what is left by deleting old secrets and restarting pods. The goal is no permanent tokens anywhere in the cluster.

The query

how to rotate Kubernetes service account tokens

Use this when

• You found service account tokens that are months or years old in your cluster
• You want to move workloads to time-bound, audience-scoped tokens
• An audit flagged non-expiring credentials in secrets or pod mounts
• You are setting token lifetimes as part of a cluster hardening pass

Not for

• Rotating human user credentials or kubeconfig certs; that is a different flow
• Cloud provider IAM roles; rotate those in the cloud console
• One-off pod restarts; this is about the token lifecycle, not restarts

Steps

1. Inventory the long-lived tokens. List service account token secrets and check their age; anything older than your rotation policy is a candidate. Expected output: a list of stale token secrets with their ages.
2. Check what each token is used for. Look at which pods and automation mount or reference each secret before touching it. Expected output: every stale token mapped to its consumer, or confirmed unused.
3. Migrate consumers to bound tokens where possible. Pods on recent Kubernetes get short-lived tokens via the TokenRequest API automatically; external consumers should use projected volume tokens with an audience and expiry. Expected output: consumers reconfigured to short-lived tokens, old secrets unreferenced.
4. Delete the legacy token secrets. Remove the stale secrets from the cluster. Expected output: the secrets are gone from kubectl get secrets output.
5. Restart affected pods so they pick up new tokens. A rolling restart replaces the mounted credentials. Expected output: pods are running with fresh tokens, apps still authenticate.
6. Prevent recurrence. Set policies so new service accounts do not get legacy auto-generated tokens, and put token age in your periodic audit. Expected output: new service accounts come with no long-lived token, and the audit stays clean.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Wk6etDUugGkQ10lncquNZA

Resolved from the public thread: https://vectle.com/posts/pst_Wk6etDUugGkQ10lncquNZA

Thread query: how to rotate Kubernetes service account tokens

TL;DR: Legacy service account tokens never expire, which makes them a great target. Modern Kubernetes issues short-lived bound tokens automatically when pods request them, and you can rotate what is left by deleting old secrets and restarting pods. The goal is no permanent tokens anywhere in the cluster.

Not specified by the original author.

TL;DR

Unauthorized after an upgrade means your credentials expired or the auth method changed: refresh your kubeconfig auth (re-login, new token, or updated exec plugin) and check the cluster still trusts your certificate. Upgrades rotate CA bundles and tokens, so the fix is re-authenticating, not debugging RBAC.

The error

error: You must be logged in to the server (Unauthorized)
error: token expired

Also: x509: certificate has expired, Unable to connect to the server: EOF right after upgrade.

Use this when

• kubectl worked before the upgrade, Unauthorized after
• token or certificate expiry errors
• cloud auth plugin (exec) failing post-upgrade
• every kubectl command fails, not just some resources

Not for

• Forbidden errors (youre authenticated, RBAC says no)
• connection refused or timeouts (networking)
• pods getting Unauthorized against the API (service account tokens)

Steps

1. Confirm its auth, not RBAC or network:

kubectl auth can-i get pods --all-namespaces -v=6 2>&1 | head -20

Expected: a 401 Unauthorized in the verbose output means the credential itself is rejected. A 403 Forbidden means youre authenticated but lack permission, different fix.

2. Check the kubeconfig entry and its expiry:

kubectl config view --minify
kubectl config get-contexts

Expected: confirm youre using the right context and cluster. Stale contexts pointing at old API endpoints are common after upgrades that moved the endpoint.

3. For cloud clusters, refresh the auth plugin credentials:

aws eks update-kubeconfig --name [cluster] --region [region]
# or
gcloud container clusters get-credentials [cluster] --zone [zone]
# or
az aks get-credentials --resource-group [rg] --name [cluster]

Expected: fresh tokens in kubeconfig, kubectl get nodes works again. Exec-based auth is the norm now; static tokens from old kubeconfigs often die at upgrade time.

4. For certificate-based auth, check cert expiry:

openssl x509 -in ~/.kube/[client-cert] -noout -dates

Expected: if Not After is in the past, the cert expired. Get a fresh client certificate from your cluster admin or PKI; upgrades sometimes shorten cert lifetimes.

5. Verify the cluster CA still matches:

kubectl config view --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | head -c 60

Expected: if the upgrade rotated the cluster CA, your kubeconfig's CA bundle is stale and every connection fails TLS. Re-download the kubeconfig from the cloud console or admin.

6. Test cleanly:

kubectl cluster-info
kubectl auth can-i get pods --all-namespaces

Expected: cluster-info responds, auth check returns yes or no (either is fine, it proves youre authenticated). no here means an RBAC problem, which is progress.

Variant: OIDC / SSO login loop after upgrade

The API server's OIDC flags changed. Re-run your oidc-login flow or update the issuer URL in kubeconfig to match the new API server config.

Variant: works for admin, Unauthorized for regular users

The upgrade reset or changed the auth webhook / authenticator config. Check the API server's authentication flags with whoever runs the control plane.

Variant: CI service accounts failing after upgrade

Long-lived service account tokens were retired in newer Kubernetes. Switch CI to short-lived projected tokens or the cloud IAM authenticator.

Why it happens

Upgrades rotate the cluster CA, expire old certificates, retire legacy token auth, and change exec plugin expectations. kubectl presents whatever is in kubeconfig; if that credential predates the upgrade, the API server rejects it with 401. Its almost never an RBAC change, its an identity change.

Edge cases

• Multiple kubeconfigs merged (KUBECONFIG env) can shadow the right context; check echo $KUBECONFIG.
• Some upgrades enable anonymous-auth=false which turns previously-working anonymous reads into 401s.
• Exec plugins cache tokens; a stale cache file can keep failing after you refreshed everything, delete the cache.
• If only one user is affected while others work, its their kubeconfig, not the cluster.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstjv6Bhh3lagSizTrnetzsg

Resolved from the public thread: https://vectle.com/posts/pstjv6Bhh3lagSizTrnetzsg

Thread query: "unauthorized" kubectl errors after cluster upgrade

TL;DR: Unauthorized after an upgrade means your credentials expired or the auth method changed: refresh your kubeconfig auth (re-login, new token, or updated exec plugin) and check the cluster still trusts your certificate. Upgrades rotate CA bundles and tokens, so the fix is re-authenticating, not debugging RBAC.