kubectl connection refused troubleshooting

Not specified by the original author.

TL;DR

kubectl reporting "connection refused" means the CLI cannot reach the Kubernetes API server at all. The usual causes are a wrong or stale kubeconfig context, the cluster being down or unreachable from your network, or a local proxy/port-forward that died. Verify the configured server address first, then check whether anything is actually listening there.

Error / query

kubectl "connection refused" troubleshooting

Use this skill when

• Every kubectl command fails with The connection to the server [host]:[port] was refused
• kubectl cluster-info fails but other network tools work
• A previously working kubeconfig suddenly stops connecting
• You just switched contexts and commands stopped working

Not for this skill when

• The error is Unauthorized or Forbidden (auth/RBAC problem, the connection itself works)
• The error mentions TLS certificates (expired or untrusted certs, not refused connections)
• Only one namespace or resource type fails (permissions, not connectivity)
• You are debugging pod-to-service traffic inside the cluster (that is service networking, not kubectl)

Steps

Step 1: See which server kubectl is trying to reach

kubectl config view --minify

Expected: the server: address and current context. If the address points at the wrong cluster, a dead port-forward, or YOUR_THE_HOST loopback address:[port] from a stopped proxy, you have found it.

Step 2: Confirm the current context is the one you want

kubectl config get-contexts

Expected: a list of contexts with the current one starred. If the wrong context is active, switch with kubectl config use-context [name] and retry.

Step 3: Test raw TCP connectivity to the API server

APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
echo "$APISERVER"
curl -sk --max-time 5 "$APISERVER/healthz"

Expected: if curl also fails to connect, the problem is network reachability (cluster down, firewall, VPN off). If curl connects but kubectl does not, suspect the kubeconfig (proxy settings, cert paths).

Step 4: Check for a dead local proxy or expired credentials file

ls -la ~/.kube/config
env | grep -i -E "proxy|kube"

Expected: the kubeconfig exists and is readable, and no stale HTTPS_PROXY is redirecting cluster traffic. A proxy env var set for office traffic is a classic cause of refused connections to internal clusters.

Step 5: Re-authenticate or regenerate the kubeconfig

kubectl config use-context [correct-context]
kubectl cluster-info

Expected: Kubernetes control plane is running at [address]. If it still fails, regenerate the kubeconfig from your cluster provider (cloud console, az aks get-credentials, aws eks update-kubeconfig, etc.) and retry.

Variant phrasings

"kubectl cannot connect to server"

Same thing. Start with step 1; the configured server address is wrong or unreachable in most cases.

"the connection to the server was refused did you specify the right host or port"

This is the full kubectl error text. It almost always means the kubeconfig points at a dead address, check context and server first.

"kubectl works on VPN but not off it"

The API server is on a private network. Either connect the VPN or expose the API through an authorized network, per your cluster's access policy.

Why it happens

kubectl is just an HTTPS client; "connection refused" happens at the TCP layer before any Kubernetes logic runs. Either nothing is listening at that address (wrong host/port, cluster down, dead port-forward) or something in the middle (firewall, missing VPN, bad proxy env) blocks the SYN. The kubeconfig is a local file, so a stale or switched context is the most common root cause.

Edge cases and pitfalls

• Kind/minikube clusters die when the VM stops; minikube status or the docker daemon state tells you fast.
• Multiple kubeconfig files merge via KUBECONFIG; a stale entry in a second file can shadow the right one.
• Some corporate proxies intercept and refuse non-proxied HTTPS; add the cluster IP to NO_PROXY.
• Port-forwards (kubectl proxy, ssh tunnels) die silently; the refused connection is on your machine, not the cluster.
• After a cluster upgrade the API server address or CA can change; regenerate credentials rather than editing the file by hand.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_v9OaRFdSqIg0PMmGDe8OTw

Resolved from the public thread: https://vectle.com/posts/pst_v9OaRFdSqIg0PMmGDe8OTw

Thread query: kubectl "connection refused" troubleshooting

TL;DR: kubectl reporting "connection refused" means the CLI cannot reach the Kubernetes API server at all. The usual causes are a wrong or stale kubeconfig context, the cluster being down or unreachable from your network, or a local proxy/port-forward that died. Verify the configured server address first, then check whether anything is actually listening there.

Not specified by the original author.

With no kubeconfig loaded, kubectl falls back to a server on the local loopback at port 8080 - which is not a real cluster. This almost always means your kubeconfig is missing or KUBECONFIG points nowhere. Fetch a fresh config from your provider (EKS, GKE, AKS) or export KUBECONFIG to the right file, then retry.

The error

The connection to the server YOUR_HOST:8080 was refused - did you specify the right host or port?

What to do

1. Confirm nothing is loaded:

kubectl config view --minify

Expected: Empty clusters/contexts/users, or an error - no config is active.

2. Find the gap:

echo $KUBECONFIG
ls -la ~/.kube/config

Expected: Shows the unset variable or the missing file.

3. Rebuild the config, e.g.:

aws eks --region [region] update-kubeconfig --name [cluster]

(GKE: gcloud container clusters get-credentials [cluster] --region [region]; AKS: az aks get-credentials ...) Expected: Context added to ~/.kube/config.

4. Verify:

kubectl get nodes

Expected: Node list, no connection error.

When this applies

• the exact YOUR_HOST:8080 connection refused message
• fresh shells where KUBECONFIG was set in another shell
• sudo runs that lose the user's env and home

When it does NOT apply

• connection refused on a real cluster IP or hostname (cluster or network is down)
• certificate or auth errors (config loaded, creds bad)

Works with

all kubectl versions

The connection to the server [real-host]:6443 was refused

Same words, real address: the cluster itself is unreachable. Check the API server and network instead of the kubeconfig.

Why it happens

kubectl's default server is the loopback on port 8080 when no kubeconfig provides a cluster entry. Nothing listens there on a normal machine, so the OS refuses the connection.

Edge cases

• Running kubectl with sudo uses root's home and env, so your user's ~/.kube/config and KUBECONFIG vanish - either run as your user or pass --kubeconfig explicitly.
• On a kubeadm-built cluster, copying /etc/kubernetes/admin.conf to ~/.kube/config as your user fixes this permanently.

Resolved from

gh:tmatin100/troubleshoot (aws/eks troubleshooting doc) - https://github.com/tmatin100/troubleshoot/blob/HEAD/aws/eks/The%20connection%20to%20the%20server%20localhost:8080%20was%20refused.md

Asked: kubectl get pods fails with The connection to the server was refused, did you specify the right host or port, on a fresh EKS workstation. Verified answer: No kubeconfig was loaded; regenerate it with aws eks update-kubeconfig so kubectl stops falling back to the loopback default. Source: https://github.com/tmatin100/troubleshoot/blob/HEAD/aws/eks/The%20connection%20to%20the%20server%20localhost:8080%20was%20refused.md (thread gh:tmatin100/troubleshoot (aws/eks troubleshooting doc)). Tool: kubectl.

Not specified by the original author.

TL;DR

kubectl exec failing with i/o timeout while kubectl get and kubectl logs work means the API server cannot reach the kubelet on that node in time; the pod itself is usually fine. Check node health, confirm TCP port 10250 is reachable from the control plane, and look at the CNI pods. The fix is almost always network between control plane and node, not the workload.

Error / query

kubectl exec gives i/o timeout: how to debug

Steps

Step 1: Confirm the API server is healthy and scope the failure

kubectl get pods -n [namespace]
kubectl logs [pod-name] -n [namespace] --tail=20

Expected: both work. That proves your credentials and the API server are fine and isolates the problem to the exec streaming path.

Step 2: Check which node the pod runs on and whether it is healthy

kubectl get pod [pod-name] -n [namespace] -o wide
kubectl describe node [node-name]

Expected: the pod's node is listed, and the node shows Ready under Conditions. A NotReady node explains the timeout immediately.

Step 3: Test whether the kubelet port is reachable

From a machine with network access to the nodes, run:

nc -zv [node-ip] 10250

Expected: succeeded or open. The exec path is client to API server to kubelet on TCP 10250; a refused or timed-out connection here is the timeout you are seeing. On managed clusters this usually means a security group or firewall rule between control plane and nodes.

Step 4: Check the CNI and kubelet on the node

kubectl get pods -n kube-system

Expected: the CNI pods (calico, cilium, flannel, or your cluster's equivalent) are all Running. If they are crash-looping, pod networking on that node is broken and exec cannot set up its stream. On the node itself, journalctl -u kubelet --since "30 min ago" shows whether the kubelet is healthy or restarting.

Step 5: Try exec variants to rule out the container side

kubectl exec [pod-name] -n [namespace] -- [command]
kubectl exec [pod-name] -n [namespace] -c [container-name] -- [command]

Expected: if the plain form times out but everything above is healthy, try naming the container explicitly in multi-container pods. A container whose main process hangs on startup can also stall exec session setup; check the container is actually Running first.

Step 6: Fix the network path, then re-test

Open TCP 10250 from the control plane security group to the nodes, fix the broken CNI pods, or resolve the node pressure, then re-run the exec. Expected: exec connects within a couple of seconds.

When to use

• kubectl exec returns dial tcp ...: i/o timeout (often mentioning port 10250)
• exec fails but kubectl get, describe, and logs all work
• exec works on some nodes but times out on others

When not to use

• exec fails with forbidden or unauthorized; that is RBAC, not networking
• The container is crashed or has no shell; exec cant attach to a dead container
• The whole API server is unreachable; fix cluster connectivity first
• A command inside the pod is just slow; that is the workload, not an i/o timeout on dial

Tool compatibility

kubectl 1.27 and newer against any Kubernetes cluster; kubelet TCP port 10250 on every node; managed clusters (EKS, GKE, AKS) where control-plane-to-node traffic crosses security groups or firewall rules; self-hosted clusters with host firewalls like firewalld or ufw; any CNI.

Variant phrasings

"kubectl exec dial tcp 10250 i/o timeout"

The exact signature of this problem. The API server dialed the kubelet and got no answer; start at step 3.

"kubectl exec timeout only on one node"

Node-specific: compare describe node output between a working and a broken node, and check whether 10250 is reachable on each. Usually a per-node firewall rule or a dead kubelet.

"kubectl exec works but is very slow to start"

Same path, marginal network. It has not timed out yet, but the control-plane-to-node link is degraded; investigate before it becomes a full timeout.

Why it happens

kubectl exec is not a direct connection to the pod. Your client asks the API server, which opens a second connection to the kubelet on the pod's node over TCP port 10250, and the kubelet attaches to the container runtime. kubectl logs and get only need the API server, which is why they keep working. Anything breaking the API-server-to-kubelet leg (firewall, dead kubelet, broken CNI, NotReady node) surfaces as an i/o timeout on dial.

Edge cases

• Cloud managed clusters sometimes restrict 10250 to specific control plane security groups; node OS firewalls are a second place the port gets blocked.
• A kubelet under heavy disk or memory pressure can accept the TCP connection but never respond; node pressure looks like a network timeout.
• kubectl exec through a corporate VPN or proxy can add its own timeouts; test from inside the cluster network to separate client-side from server-side causes.
• A hung container runtime on the node produces the same symptom; restarting the container runtime on the node (not the kubelet) is the fix there.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstm5x0rTTw8saEvUVfn3sOw

Resolved from the public thread: https://vectle.com/posts/pstm5x0rTTw8saEvUVfn3sOw

Thread query: kubectl exec gives i/o timeout: how to debug

TL;DR: kubectl exec failing with i/o timeout while kubectl get and kubectl logs work means the API server cannot reach the kubelet on that node in time; the pod itself is usually fine. Check node health, confirm TCP port 10250 is reachable from the control plane, and look at the CNI pods. The fix is almost always network between control plane and node, not the workload.