unauthorized" kubectl errors after cluster upgrade

TL;DR

Unauthorized after an upgrade means your credentials expired or the auth method changed: refresh your kubeconfig auth (re-login, new token, or updated exec plugin) and check the cluster still trusts your certificate. Upgrades rotate CA bundles and tokens, so the fix is re-authenticating, not debugging RBAC.

The error

error: You must be logged in to the server (Unauthorized)
error: token expired

Also: x509: certificate has expired, Unable to connect to the server: EOF right after upgrade.

Use this when

• kubectl worked before the upgrade, Unauthorized after
• token or certificate expiry errors
• cloud auth plugin (exec) failing post-upgrade
• every kubectl command fails, not just some resources

Not for

• Forbidden errors (youre authenticated, RBAC says no)
• connection refused or timeouts (networking)
• pods getting Unauthorized against the API (service account tokens)

Steps

1. Confirm its auth, not RBAC or network:

kubectl auth can-i get pods --all-namespaces -v=6 2>&1 | head -20

Expected: a 401 Unauthorized in the verbose output means the credential itself is rejected. A 403 Forbidden means youre authenticated but lack permission, different fix.

1. Check the kubeconfig entry and its expiry:

kubectl config view --minify
kubectl config get-contexts

Expected: confirm youre using the right context and cluster. Stale contexts pointing at old API endpoints are common after upgrades that moved the endpoint.

1. For cloud clusters, refresh the auth plugin credentials:

aws eks update-kubeconfig --name [cluster] --region [region]
# or
gcloud container clusters get-credentials [cluster] --zone [zone]
# or
az aks get-credentials --resource-group [rg] --name [cluster]

Expected: fresh tokens in kubeconfig, kubectl get nodes works again. Exec-based auth is the norm now; static tokens from old kubeconfigs often die at upgrade time.

1. For certificate-based auth, check cert expiry:

openssl x509 -in ~/.kube/[client-cert] -noout -dates

Expected: if Not After is in the past, the cert expired. Get a fresh client certificate from your cluster admin or PKI; upgrades sometimes shorten cert lifetimes.

1. Verify the cluster CA still matches:

kubectl config view --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | head -c 60

Expected: if the upgrade rotated the cluster CA, your kubeconfig's CA bundle is stale and every connection fails TLS. Re-download the kubeconfig from the cloud console or admin.

1. Test cleanly:

kubectl cluster-info
kubectl auth can-i get pods --all-namespaces

Expected: cluster-info responds, auth check returns yes or no (either is fine, it proves youre authenticated). no here means an RBAC problem, which is progress.

Variant: OIDC / SSO login loop after upgrade

The API server's OIDC flags changed. Re-run your oidc-login flow or update the issuer URL in kubeconfig to match the new API server config.

Variant: works for admin, Unauthorized for regular users

The upgrade reset or changed the auth webhook / authenticator config. Check the API server's authentication flags with whoever runs the control plane.

Variant: CI service accounts failing after upgrade

Long-lived service account tokens were retired in newer Kubernetes. Switch CI to short-lived projected tokens or the cloud IAM authenticator.

Why it happens

Upgrades rotate the cluster CA, expire old certificates, retire legacy token auth, and change exec plugin expectations. kubectl presents whatever is in kubeconfig; if that credential predates the upgrade, the API server rejects it with 401. Its almost never an RBAC change, its an identity change.

Edge cases

• Multiple kubeconfigs merged (KUBECONFIG env) can shadow the right context; check echo $KUBECONFIG.
• Some upgrades enable anonymous-auth=false which turns previously-working anonymous reads into 401s.
• Exec plugins cache tokens; a stale cache file can keep failing after you refreshed everything, delete the cache.
• If only one user is affected while others work, its their kubeconfig, not the cluster.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstjv6Bhh3lagSizTrnetzsg