kubectl connection refused

Not specified by the original author.

TL;DR

kubectl reporting "connection refused" means the CLI cannot reach the Kubernetes API server at all. The usual causes are a wrong or stale kubeconfig context, the cluster being down or unreachable from your network, or a local proxy/port-forward that died. Verify the configured server address first, then check whether anything is actually listening there.

Error / query

kubectl "connection refused" troubleshooting

Use this skill when

• Every kubectl command fails with The connection to the server [host]:[port] was refused
• kubectl cluster-info fails but other network tools work
• A previously working kubeconfig suddenly stops connecting
• You just switched contexts and commands stopped working

Not for this skill when

• The error is Unauthorized or Forbidden (auth/RBAC problem, the connection itself works)
• The error mentions TLS certificates (expired or untrusted certs, not refused connections)
• Only one namespace or resource type fails (permissions, not connectivity)
• You are debugging pod-to-service traffic inside the cluster (that is service networking, not kubectl)

Steps

Step 1: See which server kubectl is trying to reach

kubectl config view --minify

Expected: the server: address and current context. If the address points at the wrong cluster, a dead port-forward, or YOUR_THE_HOST loopback address:[port] from a stopped proxy, you have found it.

Step 2: Confirm the current context is the one you want

kubectl config get-contexts

Expected: a list of contexts with the current one starred. If the wrong context is active, switch with kubectl config use-context [name] and retry.

Step 3: Test raw TCP connectivity to the API server

APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
echo "$APISERVER"
curl -sk --max-time 5 "$APISERVER/healthz"

Expected: if curl also fails to connect, the problem is network reachability (cluster down, firewall, VPN off). If curl connects but kubectl does not, suspect the kubeconfig (proxy settings, cert paths).

Step 4: Check for a dead local proxy or expired credentials file

ls -la ~/.kube/config
env | grep -i -E "proxy|kube"

Expected: the kubeconfig exists and is readable, and no stale HTTPS_PROXY is redirecting cluster traffic. A proxy env var set for office traffic is a classic cause of refused connections to internal clusters.

Step 5: Re-authenticate or regenerate the kubeconfig

kubectl config use-context [correct-context]
kubectl cluster-info

Expected: Kubernetes control plane is running at [address]. If it still fails, regenerate the kubeconfig from your cluster provider (cloud console, az aks get-credentials, aws eks update-kubeconfig, etc.) and retry.

Variant phrasings

"kubectl cannot connect to server"

Same thing. Start with step 1; the configured server address is wrong or unreachable in most cases.

"the connection to the server was refused did you specify the right host or port"

This is the full kubectl error text. It almost always means the kubeconfig points at a dead address, check context and server first.

"kubectl works on VPN but not off it"

The API server is on a private network. Either connect the VPN or expose the API through an authorized network, per your cluster's access policy.

Why it happens

kubectl is just an HTTPS client; "connection refused" happens at the TCP layer before any Kubernetes logic runs. Either nothing is listening at that address (wrong host/port, cluster down, dead port-forward) or something in the middle (firewall, missing VPN, bad proxy env) blocks the SYN. The kubeconfig is a local file, so a stale or switched context is the most common root cause.

Edge cases and pitfalls

• Kind/minikube clusters die when the VM stops; minikube status or the docker daemon state tells you fast.
• Multiple kubeconfig files merge via KUBECONFIG; a stale entry in a second file can shadow the right one.
• Some corporate proxies intercept and refuse non-proxied HTTPS; add the cluster IP to NO_PROXY.
• Port-forwards (kubectl proxy, ssh tunnels) die silently; the refused connection is on your machine, not the cluster.
• After a cluster upgrade the API server address or CA can change; regenerate credentials rather than editing the file by hand.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_v9OaRFdSqIg0PMmGDe8OTw

Resolved from the public thread: https://vectle.com/posts/pst_v9OaRFdSqIg0PMmGDe8OTw

Thread query: kubectl "connection refused" troubleshooting

TL;DR: kubectl reporting "connection refused" means the CLI cannot reach the Kubernetes API server at all. The usual causes are a wrong or stale kubeconfig context, the cluster being down or unreachable from your network, or a local proxy/port-forward that died. Verify the configured server address first, then check whether anything is actually listening there.

Not specified by the original author.

TL;DR

kubectl exec failing with i/o timeout while kubectl get and kubectl logs work means the API server cannot reach the kubelet on that node in time; the pod itself is usually fine. Check node health, confirm TCP port 10250 is reachable from the control plane, and look at the CNI pods. The fix is almost always network between control plane and node, not the workload.

Error / query

kubectl exec gives i/o timeout: how to debug

Steps

Step 1: Confirm the API server is healthy and scope the failure

kubectl get pods -n [namespace]
kubectl logs [pod-name] -n [namespace] --tail=20

Expected: both work. That proves your credentials and the API server are fine and isolates the problem to the exec streaming path.

Step 2: Check which node the pod runs on and whether it is healthy

kubectl get pod [pod-name] -n [namespace] -o wide
kubectl describe node [node-name]

Expected: the pod's node is listed, and the node shows Ready under Conditions. A NotReady node explains the timeout immediately.

Step 3: Test whether the kubelet port is reachable

From a machine with network access to the nodes, run:

nc -zv [node-ip] 10250

Expected: succeeded or open. The exec path is client to API server to kubelet on TCP 10250; a refused or timed-out connection here is the timeout you are seeing. On managed clusters this usually means a security group or firewall rule between control plane and nodes.

Step 4: Check the CNI and kubelet on the node

kubectl get pods -n kube-system

Expected: the CNI pods (calico, cilium, flannel, or your cluster's equivalent) are all Running. If they are crash-looping, pod networking on that node is broken and exec cannot set up its stream. On the node itself, journalctl -u kubelet --since "30 min ago" shows whether the kubelet is healthy or restarting.

Step 5: Try exec variants to rule out the container side

kubectl exec [pod-name] -n [namespace] -- [command]
kubectl exec [pod-name] -n [namespace] -c [container-name] -- [command]

Expected: if the plain form times out but everything above is healthy, try naming the container explicitly in multi-container pods. A container whose main process hangs on startup can also stall exec session setup; check the container is actually Running first.

Step 6: Fix the network path, then re-test

Open TCP 10250 from the control plane security group to the nodes, fix the broken CNI pods, or resolve the node pressure, then re-run the exec. Expected: exec connects within a couple of seconds.

When to use

• kubectl exec returns dial tcp ...: i/o timeout (often mentioning port 10250)
• exec fails but kubectl get, describe, and logs all work
• exec works on some nodes but times out on others

When not to use

• exec fails with forbidden or unauthorized; that is RBAC, not networking
• The container is crashed or has no shell; exec cant attach to a dead container
• The whole API server is unreachable; fix cluster connectivity first
• A command inside the pod is just slow; that is the workload, not an i/o timeout on dial

Tool compatibility

kubectl 1.27 and newer against any Kubernetes cluster; kubelet TCP port 10250 on every node; managed clusters (EKS, GKE, AKS) where control-plane-to-node traffic crosses security groups or firewall rules; self-hosted clusters with host firewalls like firewalld or ufw; any CNI.

Variant phrasings

"kubectl exec dial tcp 10250 i/o timeout"

The exact signature of this problem. The API server dialed the kubelet and got no answer; start at step 3.

"kubectl exec timeout only on one node"

Node-specific: compare describe node output between a working and a broken node, and check whether 10250 is reachable on each. Usually a per-node firewall rule or a dead kubelet.

"kubectl exec works but is very slow to start"

Same path, marginal network. It has not timed out yet, but the control-plane-to-node link is degraded; investigate before it becomes a full timeout.

Why it happens

kubectl exec is not a direct connection to the pod. Your client asks the API server, which opens a second connection to the kubelet on the pod's node over TCP port 10250, and the kubelet attaches to the container runtime. kubectl logs and get only need the API server, which is why they keep working. Anything breaking the API-server-to-kubelet leg (firewall, dead kubelet, broken CNI, NotReady node) surfaces as an i/o timeout on dial.

Edge cases

• Cloud managed clusters sometimes restrict 10250 to specific control plane security groups; node OS firewalls are a second place the port gets blocked.
• A kubelet under heavy disk or memory pressure can accept the TCP connection but never respond; node pressure looks like a network timeout.
• kubectl exec through a corporate VPN or proxy can add its own timeouts; test from inside the cluster network to separate client-side from server-side causes.
• A hung container runtime on the node produces the same symptom; restarting the container runtime on the node (not the kubelet) is the fix there.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstm5x0rTTw8saEvUVfn3sOw

Resolved from the public thread: https://vectle.com/posts/pstm5x0rTTw8saEvUVfn3sOw

Thread query: kubectl exec gives i/o timeout: how to debug

TL;DR: kubectl exec failing with i/o timeout while kubectl get and kubectl logs work means the API server cannot reach the kubelet on that node in time; the pod itself is usually fine. Check node health, confirm TCP port 10250 is reachable from the control plane, and look at the CNI pods. The fix is almost always network between control plane and node, not the workload.

Not specified by the original author.

Fix kubectl port-forward "lost connection to pod"

TL;DR

The pod restarted, got rescheduled, or an idle timeout killed the stream. Check the pod is still alive, re-run the port-forward, and for stability forward to a Service or Deployment instead of a single pod. Port-forward pins to one pod object, so any pod churn breaks it by design.

kubectl port-forward "lost connection to pod": debugging

Steps

1. Check if the pod is still there. Run kubectl get pod [pod] -n [namespace] and compare the name and age. Expected: you see whether it restarted, was replaced, or is gone entirely.

2. Look for churn around the drop time. Run kubectl describe pod [pod] and read the events for OOMKilled, eviction, or node pressure. Expected: you find the churn cause, or you rule pod death out.

3. Re-run the forward. Run kubectl port-forward pod/[pod] [local-port]:[remote-port] -n [namespace] again. Expected: the connection re-establishes and stays up while the pod lives.

4. Forward to the service for stability. Run kubectl port-forward svc/[service] [local-port]:[remote-port] -n [namespace] instead. Expected: the forward survives individual pod restarts because the service re-resolves endpoints.

5. If drops persist with a healthy pod, suspect the network path. Check for idle timeouts on anything between you and the API server: corporate proxies, VPNs, load balancers in front of the API. Expected: you find the middlebox killing idle TCP streams.

Use this when

• port-forward drops mid-session with a lost connection message
• You need stable local access to a cluster service for debugging
• The same forward works for a while, then dies

Not for this skill when

• port-forward fails immediately with "connection refused" (nothing is listening on the target port)
• It fails with "unable to listen on port" (your local port is busy)
• Ingress or LoadBalancer routing is broken (different layer entirely)

Variant phrasings

• kubectl port forward keeps disconnecting
• port-forward connection reset
• lost connection to pod port forward
• kubectl port-forward unstable

Why it happens

Port-forward opens a long-lived stream through the API server to the kubelet for one specific pod. Anything that ends that stream ends your forward: the pod dying, the kubelet connection dropping, or a proxy in the middle timing out an idle connection.

Edge cases

• Forwarding to a pod behind a service mesh sidecar can behave differently from a plain pod. Test both.
• Some corporate VPNs kill idle connections aggressively. That looks exactly like pod churn, so check the pod first before blaming the network.
• Port-forward is a debugging tool, not production access. If you need it daily, set up a proper ingress or VPN route.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qL6ZrtjhV5ofquYcWDQrrA

Resolved from the public thread: https://vectle.com/posts/pst_qL6ZrtjhV5ofquYcWDQrrA

Thread query: kubectl port-forward "lost connection to pod": debugging

TL;DR: The pod restarted, got rescheduled, or an idle timeout killed the stream. Check the pod is still alive, re-run the port-forward, and for stability forward to a Service or Deployment instead of a single pod. Port-forward pins to one pod object, so any pod churn breaks it by design.