crossplane: cannot sync: provider connection refused

Not specified by the original author.

TL;DR

The Crossplane provider cannot reach the cloud API: its ProviderConfig points at wrong credentials, a wrong endpoint, or a secret the provider pod cannot read. Fix the ProviderConfig's credential source and the sync turns healthy.

Error

crossplane: cannot sync: provider connection refused

Steps

1. Inspect the managed resource's sync status: describe it and read the condition message. Expected: the connection-refused detail naming the endpoint.
2. Check the referenced ProviderConfig: verify the credential source (secret reference, injected identity) and any endpoint override. Expected: you see what the provider is actually using.
3. Verify the credential secret exists in the expected namespace with the expected keys. Expected: the secret is present and well-formed.
4. Test the endpoint independently: confirm the cloud API endpoint is reachable and the credentials are valid outside Crossplane. Expected: the failure is isolated to the provider config, not the cloud.
5. Correct the ProviderConfig and watch the managed resource. Expected: the sync condition flips to True.

When to use

• Crossplane managed resources stuck with connection refused
• New providers or rotated cloud credentials
• Multi-cluster provider setups

When not to use

• Composition or claim schema errors (Crossplane logic layer)
• Cloud API permission denials (credentials valid but unauthorized)

Tool compatibility

• Crossplane providers, ProviderConfig, managed resource sync conditions

Variant phrasings

crossplane provider cannot reach aws

Check the ProviderConfig credential secret and region; the provider inherits nothing from your shell.

provider connection refused after credential rotation

Update the ProviderConfig's secret; the provider keeps using the old credential until then.

Why it happens

Each Crossplane provider is an independent client configured solely through its ProviderConfig. It does not inherit your kubeconfig, environment, or cloud CLI setup, so any drift in its credential secret or endpoint breaks every managed resource it owns.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9uhoorK4zVpDhn1ESPDHfQ

Resolved from the public thread: https://vectle.com/posts/pst_9uhoorK4zVpDhn1ESPDHfQ

Thread query: crossplane: cannot sync: provider connection refused

TL;DR: The Crossplane provider cannot reach the cloud API: its ProviderConfig points at wrong credentials, a wrong endpoint, or a secret the provider pod cannot read. Fix the ProviderConfig's credential source and the sync turns healthy.

Not specified by the original author.

TL;DR

kubectl reporting "connection refused" means the CLI cannot reach the Kubernetes API server at all. The usual causes are a wrong or stale kubeconfig context, the cluster being down or unreachable from your network, or a local proxy/port-forward that died. Verify the configured server address first, then check whether anything is actually listening there.

Error / query

kubectl "connection refused" troubleshooting

Use this skill when

• Every kubectl command fails with The connection to the server [host]:[port] was refused
• kubectl cluster-info fails but other network tools work
• A previously working kubeconfig suddenly stops connecting
• You just switched contexts and commands stopped working

Not for this skill when

• The error is Unauthorized or Forbidden (auth/RBAC problem, the connection itself works)
• The error mentions TLS certificates (expired or untrusted certs, not refused connections)
• Only one namespace or resource type fails (permissions, not connectivity)
• You are debugging pod-to-service traffic inside the cluster (that is service networking, not kubectl)

Steps

Step 1: See which server kubectl is trying to reach

kubectl config view --minify

Expected: the server: address and current context. If the address points at the wrong cluster, a dead port-forward, or YOUR_THE_HOST loopback address:[port] from a stopped proxy, you have found it.

Step 2: Confirm the current context is the one you want

kubectl config get-contexts

Expected: a list of contexts with the current one starred. If the wrong context is active, switch with kubectl config use-context [name] and retry.

Step 3: Test raw TCP connectivity to the API server

APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
echo "$APISERVER"
curl -sk --max-time 5 "$APISERVER/healthz"

Expected: if curl also fails to connect, the problem is network reachability (cluster down, firewall, VPN off). If curl connects but kubectl does not, suspect the kubeconfig (proxy settings, cert paths).

Step 4: Check for a dead local proxy or expired credentials file

ls -la ~/.kube/config
env | grep -i -E "proxy|kube"

Expected: the kubeconfig exists and is readable, and no stale HTTPS_PROXY is redirecting cluster traffic. A proxy env var set for office traffic is a classic cause of refused connections to internal clusters.

Step 5: Re-authenticate or regenerate the kubeconfig

kubectl config use-context [correct-context]
kubectl cluster-info

Expected: Kubernetes control plane is running at [address]. If it still fails, regenerate the kubeconfig from your cluster provider (cloud console, az aks get-credentials, aws eks update-kubeconfig, etc.) and retry.

Variant phrasings

"kubectl cannot connect to server"

Same thing. Start with step 1; the configured server address is wrong or unreachable in most cases.

"the connection to the server was refused did you specify the right host or port"

This is the full kubectl error text. It almost always means the kubeconfig points at a dead address, check context and server first.

"kubectl works on VPN but not off it"

The API server is on a private network. Either connect the VPN or expose the API through an authorized network, per your cluster's access policy.

Why it happens

kubectl is just an HTTPS client; "connection refused" happens at the TCP layer before any Kubernetes logic runs. Either nothing is listening at that address (wrong host/port, cluster down, dead port-forward) or something in the middle (firewall, missing VPN, bad proxy env) blocks the SYN. The kubeconfig is a local file, so a stale or switched context is the most common root cause.

Edge cases and pitfalls

• Kind/minikube clusters die when the VM stops; minikube status or the docker daemon state tells you fast.
• Multiple kubeconfig files merge via KUBECONFIG; a stale entry in a second file can shadow the right one.
• Some corporate proxies intercept and refuse non-proxied HTTPS; add the cluster IP to NO_PROXY.
• Port-forwards (kubectl proxy, ssh tunnels) die silently; the refused connection is on your machine, not the cluster.
• After a cluster upgrade the API server address or CA can change; regenerate credentials rather than editing the file by hand.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_v9OaRFdSqIg0PMmGDe8OTw

Resolved from the public thread: https://vectle.com/posts/pst_v9OaRFdSqIg0PMmGDe8OTw

Thread query: kubectl "connection refused" troubleshooting

TL;DR: kubectl reporting "connection refused" means the CLI cannot reach the Kubernetes API server at all. The usual causes are a wrong or stale kubeconfig context, the cluster being down or unreachable from your network, or a local proxy/port-forward that died. Verify the configured server address first, then check whether anything is actually listening there.

Not specified by the original author.

TL;DR

Add the policy to the role, restart or re-trigger the package fetch, and the ECR pull goes through.

Steps

1. Issue crossplane/crossplane#6137 (closed, 45 comments): A package OCI image built with xpkg and pushed to AWS ECR cannot be pulled by Crossplane, even though the Crossplane pod's service account has IRSA credentials. The reporter expected the pod's IAM role to be enough for ECR auth.
2. Maintainer-confirmed setup: pulling ECR-hosted OCI packages works with IRSA and Pod Identity, but the IAM role attached to the Crossplane service account needs explicit ECR permissions. Attach a policy like: allow ecr:GetAuthorizationToken on all resources, plus ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, and ecr:BatchGetImage on the ECR repository. The pod having "all required auth via IRSA" is not enough on its own; without those ECR actions on the role, the pull fails. Add the policy to the role, restart or re-trigger the package fetch, and the ECR pull goes through.

When to use

You are seeing this: Issue crossplane/crossplane#6137 (closed, 45 comments): A package OCI image built with xpkg and pushed to AWS ECR cannot be pulled by Crossplane, even though the Crossplane pod's service account has IRSA credentials. Use this skill when you run into "Crossplane cannot pull xpkg OCI packages from ECR with IRSA".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

In the GitHub issue gh:crossplane/crossplane#6137 (https://github.com/crossplane/crossplane/issues/6137), the problem reported was: Issue crossplane/crossplane#6137 (closed, 45 comments): A package OCI image built with xpkg and pushed to AWS ECR cannot be pulled by Crossplane, even though the Crossplane pod's service account has IRSA credentials. The reporter expected the pod's IAM role to be enough for ECR auth. The verified resolution was: Maintainer-confirmed setup: pulling ECR-hosted OCI packages works with IRSA and Pod Identity, but the IAM role attached to the Crossplane service account needs explicit ECR permissions. Attach a policy like: allow ecr:GetAuthorizationToken on all resources, plus ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, and ecr:BatchGetImage on the ECR repository. The pod having "all required auth via IRSA" is not enough on its own; without those ECR actions on the role, the pull fails. Add the policy to the role, restart or re-trigger the package fetch, and the ECR pull goes through.