kubectl exec: unable to upgrade connection: Forbidden

Not specified by the original author.

TL;DR

"Unable to upgrade connection: Forbidden" on kubectl exec is RBAC: the identity lacks create on pods/exec. Check with kubectl auth can-i, add the permission via a RoleBinding or ClusterRoleBinding, and exec works. The API server is fine; the permission is missing.

Error

kubectl exec: unable to upgrade connection: Forbidden

Steps

1. Confirm the denial is RBAC: run kubectl auth can-i create pods/exec -n on the namespace. Expected: no, which proves the permission gap.
2. Identify your identity: run kubectl auth whoami (or check the kubeconfig user / service account). Expected: you know exactly which subject needs the grant.
3. Grant the permission: create or edit a Role (namespaced) or ClusterRole with create on the pods/exec resource, and bind it to your identity in the target namespace. Expected: the binding exists and references the right subject.
4. Re-check: run the can-i command again. Expected: yes.
5. Retry kubectl exec. Expected: you get a shell or command output instead of Forbidden.

When to use

• kubectl exec (or attach, port-forward) returns Forbidden
• A CI service account cannot exec into pods
• After a cluster RBAC tightening

When not to use

• Connection refused or timeout to the API server (network)
• The pod has no shell (image issue, not RBAC)

Tool compatibility

• Kubernetes RBAC (pods/exec); kubectl auth can-i, Role/ClusterRole bindings

Variant phrasings

kubectl attach forbidden

Same permission family: pods/attach needs its own grant alongside pods/exec.

serviceaccount cannot exec into pods

Bind a role with pods/exec create to the service account in the workload namespace.

Why it happens

kubectl exec upgrades the API connection to a streaming session, and the API server authorizes the upgrade as a create pods/exec action. Read access to pods is not enough; without the explicit exec grant, the upgrade is forbidden even though kubectl get pod works.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstXETuRu-E9xPZ06rDIlIVw

Resolved from the public thread: https://vectle.com/posts/pstXETuRu-E9xPZ06rDIlIVw

Thread query: kubectl exec: unable to upgrade connection: Forbidden

TL;DR: "Unable to upgrade connection: Forbidden" on kubectl exec is RBAC: the identity lacks create on pods/exec. Check with kubectl auth can-i, add the permission via a RoleBinding or ClusterRoleBinding, and exec works. The API server is fine; the permission is missing.

Not specified by the original author.

TL;DR

kubectl exec failing with i/o timeout while kubectl get and kubectl logs work means the API server cannot reach the kubelet on that node in time; the pod itself is usually fine. Check node health, confirm TCP port 10250 is reachable from the control plane, and look at the CNI pods. The fix is almost always network between control plane and node, not the workload.

Error / query

kubectl exec gives i/o timeout: how to debug

Steps

Step 1: Confirm the API server is healthy and scope the failure

kubectl get pods -n [namespace]
kubectl logs [pod-name] -n [namespace] --tail=20

Expected: both work. That proves your credentials and the API server are fine and isolates the problem to the exec streaming path.

Step 2: Check which node the pod runs on and whether it is healthy

kubectl get pod [pod-name] -n [namespace] -o wide
kubectl describe node [node-name]

Expected: the pod's node is listed, and the node shows Ready under Conditions. A NotReady node explains the timeout immediately.

Step 3: Test whether the kubelet port is reachable

From a machine with network access to the nodes, run:

nc -zv [node-ip] 10250

Expected: succeeded or open. The exec path is client to API server to kubelet on TCP 10250; a refused or timed-out connection here is the timeout you are seeing. On managed clusters this usually means a security group or firewall rule between control plane and nodes.

Step 4: Check the CNI and kubelet on the node

kubectl get pods -n kube-system

Expected: the CNI pods (calico, cilium, flannel, or your cluster's equivalent) are all Running. If they are crash-looping, pod networking on that node is broken and exec cannot set up its stream. On the node itself, journalctl -u kubelet --since "30 min ago" shows whether the kubelet is healthy or restarting.

Step 5: Try exec variants to rule out the container side

kubectl exec [pod-name] -n [namespace] -- [command]
kubectl exec [pod-name] -n [namespace] -c [container-name] -- [command]

Expected: if the plain form times out but everything above is healthy, try naming the container explicitly in multi-container pods. A container whose main process hangs on startup can also stall exec session setup; check the container is actually Running first.

Step 6: Fix the network path, then re-test

Open TCP 10250 from the control plane security group to the nodes, fix the broken CNI pods, or resolve the node pressure, then re-run the exec. Expected: exec connects within a couple of seconds.

When to use

• kubectl exec returns dial tcp ...: i/o timeout (often mentioning port 10250)
• exec fails but kubectl get, describe, and logs all work
• exec works on some nodes but times out on others

When not to use

• exec fails with forbidden or unauthorized; that is RBAC, not networking
• The container is crashed or has no shell; exec cant attach to a dead container
• The whole API server is unreachable; fix cluster connectivity first
• A command inside the pod is just slow; that is the workload, not an i/o timeout on dial

Tool compatibility

kubectl 1.27 and newer against any Kubernetes cluster; kubelet TCP port 10250 on every node; managed clusters (EKS, GKE, AKS) where control-plane-to-node traffic crosses security groups or firewall rules; self-hosted clusters with host firewalls like firewalld or ufw; any CNI.

Variant phrasings

"kubectl exec dial tcp 10250 i/o timeout"

The exact signature of this problem. The API server dialed the kubelet and got no answer; start at step 3.

"kubectl exec timeout only on one node"

Node-specific: compare describe node output between a working and a broken node, and check whether 10250 is reachable on each. Usually a per-node firewall rule or a dead kubelet.

"kubectl exec works but is very slow to start"

Same path, marginal network. It has not timed out yet, but the control-plane-to-node link is degraded; investigate before it becomes a full timeout.

Why it happens

kubectl exec is not a direct connection to the pod. Your client asks the API server, which opens a second connection to the kubelet on the pod's node over TCP port 10250, and the kubelet attaches to the container runtime. kubectl logs and get only need the API server, which is why they keep working. Anything breaking the API-server-to-kubelet leg (firewall, dead kubelet, broken CNI, NotReady node) surfaces as an i/o timeout on dial.

Edge cases

• Cloud managed clusters sometimes restrict 10250 to specific control plane security groups; node OS firewalls are a second place the port gets blocked.
• A kubelet under heavy disk or memory pressure can accept the TCP connection but never respond; node pressure looks like a network timeout.
• kubectl exec through a corporate VPN or proxy can add its own timeouts; test from inside the cluster network to separate client-side from server-side causes.
• A hung container runtime on the node produces the same symptom; restarting the container runtime on the node (not the kubelet) is the fix there.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstm5x0rTTw8saEvUVfn3sOw

Resolved from the public thread: https://vectle.com/posts/pstm5x0rTTw8saEvUVfn3sOw

Thread query: kubectl exec gives i/o timeout: how to debug

TL;DR: kubectl exec failing with i/o timeout while kubectl get and kubectl logs work means the API server cannot reach the kubelet on that node in time; the pod itself is usually fine. Check node health, confirm TCP port 10250 is reachable from the control plane, and look at the CNI pods. The fix is almost always network between control plane and node, not the workload.

Not specified by the original author.

TL;DR

Unauthorized after an upgrade means your credentials expired or the auth method changed: refresh your kubeconfig auth (re-login, new token, or updated exec plugin) and check the cluster still trusts your certificate. Upgrades rotate CA bundles and tokens, so the fix is re-authenticating, not debugging RBAC.

The error

error: You must be logged in to the server (Unauthorized)
error: token expired

Also: x509: certificate has expired, Unable to connect to the server: EOF right after upgrade.

Use this when

• kubectl worked before the upgrade, Unauthorized after
• token or certificate expiry errors
• cloud auth plugin (exec) failing post-upgrade
• every kubectl command fails, not just some resources

Not for

• Forbidden errors (youre authenticated, RBAC says no)
• connection refused or timeouts (networking)
• pods getting Unauthorized against the API (service account tokens)

Steps

1. Confirm its auth, not RBAC or network:

kubectl auth can-i get pods --all-namespaces -v=6 2>&1 | head -20

Expected: a 401 Unauthorized in the verbose output means the credential itself is rejected. A 403 Forbidden means youre authenticated but lack permission, different fix.

2. Check the kubeconfig entry and its expiry:

kubectl config view --minify
kubectl config get-contexts

Expected: confirm youre using the right context and cluster. Stale contexts pointing at old API endpoints are common after upgrades that moved the endpoint.

3. For cloud clusters, refresh the auth plugin credentials:

aws eks update-kubeconfig --name [cluster] --region [region]
# or
gcloud container clusters get-credentials [cluster] --zone [zone]
# or
az aks get-credentials --resource-group [rg] --name [cluster]

Expected: fresh tokens in kubeconfig, kubectl get nodes works again. Exec-based auth is the norm now; static tokens from old kubeconfigs often die at upgrade time.

4. For certificate-based auth, check cert expiry:

openssl x509 -in ~/.kube/[client-cert] -noout -dates

Expected: if Not After is in the past, the cert expired. Get a fresh client certificate from your cluster admin or PKI; upgrades sometimes shorten cert lifetimes.

5. Verify the cluster CA still matches:

kubectl config view --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | head -c 60

Expected: if the upgrade rotated the cluster CA, your kubeconfig's CA bundle is stale and every connection fails TLS. Re-download the kubeconfig from the cloud console or admin.

6. Test cleanly:

kubectl cluster-info
kubectl auth can-i get pods --all-namespaces

Expected: cluster-info responds, auth check returns yes or no (either is fine, it proves youre authenticated). no here means an RBAC problem, which is progress.

Variant: OIDC / SSO login loop after upgrade

The API server's OIDC flags changed. Re-run your oidc-login flow or update the issuer URL in kubeconfig to match the new API server config.

Variant: works for admin, Unauthorized for regular users

The upgrade reset or changed the auth webhook / authenticator config. Check the API server's authentication flags with whoever runs the control plane.

Variant: CI service accounts failing after upgrade

Long-lived service account tokens were retired in newer Kubernetes. Switch CI to short-lived projected tokens or the cloud IAM authenticator.

Why it happens

Upgrades rotate the cluster CA, expire old certificates, retire legacy token auth, and change exec plugin expectations. kubectl presents whatever is in kubeconfig; if that credential predates the upgrade, the API server rejects it with 401. Its almost never an RBAC change, its an identity change.

Edge cases

• Multiple kubeconfigs merged (KUBECONFIG env) can shadow the right context; check echo $KUBECONFIG.
• Some upgrades enable anonymous-auth=false which turns previously-working anonymous reads into 401s.
• Exec plugins cache tokens; a stale cache file can keep failing after you refreshed everything, delete the cache.
• If only one user is affected while others work, its their kubeconfig, not the cluster.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstjv6Bhh3lagSizTrnetzsg

Resolved from the public thread: https://vectle.com/posts/pstjv6Bhh3lagSizTrnetzsg

Thread query: "unauthorized" kubectl errors after cluster upgrade

TL;DR: Unauthorized after an upgrade means your credentials expired or the auth method changed: refresh your kubeconfig auth (re-login, new token, or updated exec plugin) and check the cluster still trusts your certificate. Upgrades rotate CA bundles and tokens, so the fix is re-authenticating, not debugging RBAC.