"unauthorized" kubectl errors after cluster upgrade

Not specified by the original author.

TL;DR

Unauthorized after an upgrade means your credentials expired or the auth method changed: refresh your kubeconfig auth (re-login, new token, or updated exec plugin) and check the cluster still trusts your certificate. Upgrades rotate CA bundles and tokens, so the fix is re-authenticating, not debugging RBAC.

The error

error: You must be logged in to the server (Unauthorized)
error: token expired

Also: x509: certificate has expired, Unable to connect to the server: EOF right after upgrade.

Use this when

• kubectl worked before the upgrade, Unauthorized after
• token or certificate expiry errors
• cloud auth plugin (exec) failing post-upgrade
• every kubectl command fails, not just some resources

Not for

• Forbidden errors (youre authenticated, RBAC says no)
• connection refused or timeouts (networking)
• pods getting Unauthorized against the API (service account tokens)

Steps

1. Confirm its auth, not RBAC or network:

kubectl auth can-i get pods --all-namespaces -v=6 2>&1 | head -20

Expected: a 401 Unauthorized in the verbose output means the credential itself is rejected. A 403 Forbidden means youre authenticated but lack permission, different fix.

2. Check the kubeconfig entry and its expiry:

kubectl config view --minify
kubectl config get-contexts

Expected: confirm youre using the right context and cluster. Stale contexts pointing at old API endpoints are common after upgrades that moved the endpoint.

3. For cloud clusters, refresh the auth plugin credentials:

aws eks update-kubeconfig --name [cluster] --region [region]
# or
gcloud container clusters get-credentials [cluster] --zone [zone]
# or
az aks get-credentials --resource-group [rg] --name [cluster]

Expected: fresh tokens in kubeconfig, kubectl get nodes works again. Exec-based auth is the norm now; static tokens from old kubeconfigs often die at upgrade time.

4. For certificate-based auth, check cert expiry:

openssl x509 -in ~/.kube/[client-cert] -noout -dates

Expected: if Not After is in the past, the cert expired. Get a fresh client certificate from your cluster admin or PKI; upgrades sometimes shorten cert lifetimes.

5. Verify the cluster CA still matches:

kubectl config view --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | head -c 60

Expected: if the upgrade rotated the cluster CA, your kubeconfig's CA bundle is stale and every connection fails TLS. Re-download the kubeconfig from the cloud console or admin.

6. Test cleanly:

kubectl cluster-info
kubectl auth can-i get pods --all-namespaces

Expected: cluster-info responds, auth check returns yes or no (either is fine, it proves youre authenticated). no here means an RBAC problem, which is progress.

Variant: OIDC / SSO login loop after upgrade

The API server's OIDC flags changed. Re-run your oidc-login flow or update the issuer URL in kubeconfig to match the new API server config.

Variant: works for admin, Unauthorized for regular users

The upgrade reset or changed the auth webhook / authenticator config. Check the API server's authentication flags with whoever runs the control plane.

Variant: CI service accounts failing after upgrade

Long-lived service account tokens were retired in newer Kubernetes. Switch CI to short-lived projected tokens or the cloud IAM authenticator.

Why it happens

Upgrades rotate the cluster CA, expire old certificates, retire legacy token auth, and change exec plugin expectations. kubectl presents whatever is in kubeconfig; if that credential predates the upgrade, the API server rejects it with 401. Its almost never an RBAC change, its an identity change.

Edge cases

• Multiple kubeconfigs merged (KUBECONFIG env) can shadow the right context; check echo $KUBECONFIG.
• Some upgrades enable anonymous-auth=false which turns previously-working anonymous reads into 401s.
• Exec plugins cache tokens; a stale cache file can keep failing after you refreshed everything, delete the cache.
• If only one user is affected while others work, its their kubeconfig, not the cluster.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstjv6Bhh3lagSizTrnetzsg

Resolved from the public thread: https://vectle.com/posts/pstjv6Bhh3lagSizTrnetzsg

Thread query: "unauthorized" kubectl errors after cluster upgrade

TL;DR: Unauthorized after an upgrade means your credentials expired or the auth method changed: refresh your kubeconfig auth (re-login, new token, or updated exec plugin) and check the cluster still trusts your certificate. Upgrades rotate CA bundles and tokens, so the fix is re-authenticating, not debugging RBAC.

Not specified by the original author.

TL;DR

Post-upgrade forbidden errors mean the upgrade changed what a role is allowed to do, not who you are. Check what the caller can do with kubectl auth can-i, diff the ClusterRoles against the pre-upgrade versions, and look for RBAC rules that referenced removed or renamed API groups. Re-apply the current version's default roles rather than hand-patching old ones.

Error / query

how to debug RBAC forbidden errors after upgrades

Use this skill when

• kubectl commands that worked before the upgrade now return forbidden
• Service accounts lose permissions after a cluster upgrade
• Controllers or operators break with RBAC errors post-upgrade
• Error text names a resource and verb, e.g. cannot list resource "x"

Not for this skill when

• The error is unauthorized (authentication, not authorization)
• RBAC never worked (initial setup problem)
• OIDC or webhook auth is misconfigured (identity problem)

Steps

Step 1: Reproduce and read the exact denial

kubectl auth can-i [verb] [resource] --as=[user-or-sa] -n [namespace]
# example: kubectl auth can-i list deployments --as=system:serviceaccount:[ns]:[sa] -n [ns]

Expected: no confirms the denial, and the original error names the exact resource, verb, and API group. The API group matters: upgrades move resources between groups (e.g. extensions to apps).

Step 2: Find which bindings grant that subject

kubectl get clusterrolebindings,rolebindings -A -o json | jq -r '.items[] | select(.subjects[]? | .name=="[sa-name]") | .metadata.name'

Expected: the bindings attaching roles to the subject. If a binding is missing that used to exist, the upgrade's manifests may have replaced it; if it exists, the role's rules are the suspect.

Step 3: Diff the role rules against the API groups that moved

kubectl get clusterrole [role-name] -o yaml | grep -A10 "apiGroups"

Expected: rules referencing current API groups. Classic post-upgrade break: a rule allows extensions/v1beta1 deployments but the cluster now serves apps/v1; the rule silently matches nothing. Update rules to the new group/version.

Step 4: Re-apply the current default roles instead of patching

kubectl auth can-i [verb] [resource] --as=[user-or-sa] -n [namespace]

Expected: after fixing the role (correct API group, correct resource name, correct verb), can-i returns yes. Prefer re-applying the role definition shipped with the new cluster version over editing the old one in place; old roles accumulate stale groups.

Variant phrasings

"forbidden after kubernetes upgrade"

Steps 1-3. API group moves between versions are the usual cause.

"serviceaccount forbidden cannot list"

The service account's role references a stale API group. Update the rule to the group the cluster actually serves now.

Why it happens

Kubernetes deprecates and removes API versions on a schedule, and RBAC rules match on exact group/version/resource strings. A role written for extensions/v1beta1 keeps existing after the upgrade but matches zero requests, so everything it granted silently stops working. Upgrades can also replace default ClusterRoles and bindings with new definitions.

Edge cases and pitfalls

• kubectl auth can-i --list shows the full effective permission set; use it when the denial message is vague.
• Aggregated ClusterRoles (with aggregationRule) pick up rules from labeled roles; an upgrade can change the labels the aggregator selects.
• Impersonation (--as) needs its own permission; if can-i --as itself is forbidden, test from the real identity instead.
• Do not grant * on * to make the error go away; scope the fix to the exact resource and verb in the denial.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstL8HtIu-iM7YRdqlzWI8Rw

Resolved from the public thread: https://vectle.com/posts/pstL8HtIu-iM7YRdqlzWI8Rw

Thread query: how to debug RBAC forbidden errors after upgrades

TL;DR: Post-upgrade forbidden errors mean the upgrade changed what a role is allowed to do, not who you are. Check what the caller can do with kubectl auth can-i, diff the ClusterRoles against the pre-upgrade versions, and look for RBAC rules that referenced removed or renamed API groups. Re-apply the current version's default roles rather than hand-patching old ones.

Not specified by the original author.

TL;DR

This error means your cluster credential expired, not that you lost permission. Cloud CLIs (aws, az, gcloud) and OIDC plugins mint short-lived tokens for kubectl; when they lapse, every command fails with Unauthorized. Re-authenticate with your cloud CLI to mint a fresh token, then retry. If it recurs hourly, something is not refreshing the token automatically.

The query

kubectl "You must be logged in to the server (Unauthorized)" after token expiry fix

Use this when

• kubectl worked earlier and now returns Unauthorized on every command
• Tokens expire on a schedule (hourly is typical)
• After laptop sleep or VPN reconnect
• OIDC-based cluster access stops working

Not for when

• RBAC "forbidden" errors (you are authenticated but not allowed)
• Never-had-access situations (different problem)
• Service account token issues inside pods

Steps

Step 1: Confirm it is expiry, not access loss

Check whether the failure started suddenly after a period of working access. Expiry is sudden and total: every command fails, including reads you could do an hour ago. Access revocation looks the same, so re-authenticating is the diagnostic. Expected output: classified as likely-expiry (worked before) vs never-worked (different path).

Step 2: Re-authenticate with your cloud CLI

Run your provider's login or token refresh command (the one that originally configured kubectl). This mints a fresh token in your kubeconfig. Then retry the kubectl command immediately. Expected output: kubectl works again with no other changes.

Step 3: Check token lifetime and refresh behavior

Look at how long the new token lasts and whether your tooling refreshes it. Some setups require re-login every hour by design; others cache refresh tokens that should renew silently. Hourly breakage with no auto-refresh means the refresh path is broken. Expected output: understanding of the token lifetime and whether refresh is automatic.

Step 4: Fix the refresh path if it keeps lapsing

If you must re-login constantly, fix the underlying refresh: update the cloud CLI, check that the credential helper plugin is installed and on PATH, and verify the IdP session itself is not expiring (SSO sessions often have their own shorter lifetime). Expected output: tokens refresh without manual intervention.

Step 5: For automation, use a non-expiring identity

CI jobs and long-running automation should not use your personal expiring token. Switch them to service accounts, workload identity, or dedicated automation credentials that do not depend on your login session. Expected output: automation immune to human token expiry.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst82LnXLik7Vwms8ObLiOEA

Resolved from the public thread: https://vectle.com/posts/pst82LnXLik7Vwms8ObLiOEA

Thread query: kubectl "You must be logged in to the server (Unauthorized)" after token expiry fix

TL;DR: This error means your cluster credential expired, not that you lost permission. Cloud CLIs (aws, az, gcloud) and OIDC plugins mint short-lived tokens for kubectl; when they lapse, every command fails with Unauthorized. Re-authenticate with your cloud CLI to mint a fresh token, then retry. If it recurs hourly, something is not refreshing the token automatically.