context deadline exceeded kubernetes

Not specified by the original author.

TL;DR

Context deadline exceeded means the API call took longer than the client's timeout, and the cause is usually an overloaded API server or slow etcd, not your command. Check API server latency and etcd health first, then look at whats hammering the API (a runaway controller or a huge list call). Fix the load and the timeouts stop.

The error

Error from server: context deadline exceeded
Unable to connect to the server: context deadline exceeded

Controllers log: failed to list *v1.Pod: context deadline exceeded.

Use this when

• kubectl commands time out intermittently
• controllers or operators log deadline exceeded
• API feels slow but isnt down
• list or watch calls hang

Not for

• connection refused (API server not listening)
• 401/403 errors (auth, not timeouts)
• timeouts from inside pods to services (different path)

Steps

1. Check if its you or the server. Time a trivial call:

time kubectl get --raw /healthz

Expected: responds in well under a second. If even /healthz is slow, the API server itself is struggling; if its fast, the problem is specific expensive calls.

2. Look at API server and etcd health:

kubectl get --raw /healthz/etcd
kubectl top pods -n kube-system -l component=kube-apiserver

Expected: etcd healthy, apiserver CPU sane. etcd slowness is the classic root cause: every API write waits on etcd quorum.

3. Find whos hammering the API:

kubectl get --raw /metrics | grep -i apiserver_request_total | sort -t ' ' -k2 -rn | head -10

Expected: the top request sources. A controller listing all pods cluster-wide every 30 seconds, or a broken operator in a hot loop, shows up immediately.

4. Check for expensive list calls without limits:

kubectl get pods -A --chunk-size=500 --request-timeout=30s | head

Expected: works fine with chunking. Code that lists everything at once (no pagination, no field selectors) times out as the cluster grows; fix the client to page and filter.

5. Raise the client timeout as a stopgap, fix the server as the cure:

kubectl get pods -A --request-timeout=60s

Expected: the call completes. But dont stop here: sustained deadline exceeded means the control plane needs help (more apiserver CPU, etcd on faster disks, fewer abusive clients).

6. Confirm recovery:

time kubectl get pods -A --request-timeout=30s

Expected: completes in seconds, no deadline errors. Watch for a full day; timeouts that return point at a recurring load pattern like a cronjob that lists everything.

Variant: only watch streams stall

Long watches timing out is often a proxy or load balancer idle timeout killing the connection, not the API server. Check LB timeouts in front of the API.

Variant: deadline exceeded right after a cluster upgrade

New admission webhooks or aggregated API servers added latency to every call. Check webhook timeouts and the health of extension API servers.

Variant: one controller logs it constantly, everything else fine

That controller's client is misconfigured (too-short timeout, no retries, huge list scope). Fix the controller, not the cluster.

Why it happens

Every API call carries a deadline. When etcd is slow, the apiserver is CPU-starved, or a client makes enormous unpaginated list calls, the work exceeds the deadline and the client gives up. The error blames the context, but the fault is almost always server-side load or client-side greed.

Edge cases

• etcd on slow disks is the silent killer; watch etcd disk wal fsync durations, not just API latency.
• Huge CRDs with thousands of objects make list calls expensive; paginate and use label selectors.
• Mutating webhooks that call out to slow services add their latency to every matching API call.
• Cloud load balancers in front of the API server have their own idle timeouts that masquerade as deadline exceeded on watches.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_L4wpKrRcfYW5s8dKND9GlQ

Resolved from the public thread: https://vectle.com/posts/pst_L4wpKrRcfYW5s8dKND9GlQ

Thread query: "context deadline exceeded" in Kubernetes API calls

TL;DR: Context deadline exceeded means the API call took longer than the client's timeout, and the cause is usually an overloaded API server or slow etcd, not your command. Check API server latency and etcd health first, then look at whats hammering the API (a runaway controller or a huge list call). Fix the load and the timeouts stop.

Not specified by the original author.

TL;DR

Most context deadline exceeded errors from kubectl or controllers are client-side timeouts firing before a slow API server answers. Raise the client timeout, add retries with backoff in controllers, and check whether client-side rate limiting is throttling you before blaming the API server. Tune the client to the server's real latency, not to defaults meant for fast local clusters.

Error / query

how to tune Kubernetes API client timeouts and retries

Use this skill when

• kubectl or controllers report context deadline exceeded
• API calls flap or time out under load
• Automation is slow and client-side throttling is suspected
• Controllers log rate-limiter waits

Not for this skill when

• The API server itself is slow or overloaded (server-side tuning)
• etcd is the bottleneck (different diagnosis)
• Requests are denied by RBAC (authorization, not timeouts)

Steps

Step 1: Measure whether the server or the client is slow

time kubectl get pods -n [namespace] --request-timeout=60s

Expected: the wall time tells you the server's real latency. If --request-timeout=60s succeeds where the default (30s) failed, the client timeout was the problem, not the server.

Step 2: Set a sane request timeout for interactive use

kubectl get pods -n [namespace] --request-timeout=2m

Expected: long list/watch calls complete instead of timing out. For scripts, export it per-command rather than relying on the default; large list calls against big clusters legitimately take over 30s.

Step 3: Tune client-go rate limiting in controllers

config.QPS = 50
config.Burst = 100

Expected: in controller code, raising QPS/Burst from the client-go defaults (5/10) stops the client from self-throttling. Watch controller logs for Waited for messages from the rate limiter; their disappearance confirms the fix.

Step 4: Add retries with backoff around API calls

for i in 1 2 3 4 5; do kubectl get [resource] [name] -n [namespace] && break || sleep $((2 ** i)); done

Expected: transient timeouts succeed on retry with exponential backoff. For controllers, use the client-go retry utilities rather than failing the reconcile loop on the first timeout.

Variant phrasings

"kubectl context deadline exceeded"

Step 1 first: raise --request-timeout and see if the call succeeds. If it does, it was the client.

"kubernetes client throttling"

Check controller logs for rate-limiter waits, then raise QPS/Burst as in step 3.

Why it happens

Client defaults assume a fast, nearby API server: 30s request timeouts and aggressive client-side rate limits. Against large, loaded, or distant clusters, legitimate calls exceed those defaults, and the client reports a timeout that looks like a server failure. The fix is matching client patience to server reality.

Edge cases and pitfalls

• Raising timeouts masks a genuinely sick API server; if p99 latency keeps climbing, tune the server instead of the client.
• Watch calls with no timeout can hang forever on network partitions; always bound watches, even generous ones.
• Very high QPS from many controllers can DDoS a small API server; raise limits gradually and watch API server latency.
• --request-timeout=0 means no timeout, which is dangerous in scripts; always set an explicit bound.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jf7Qv9jBV9iWwtT-gg-2fA

Resolved from the public thread: https://vectle.com/posts/pst_jf7Qv9jBV9iWwtT-gg-2fA

Thread query: how to tune Kubernetes API client timeouts and retries

TL;DR: Most context deadline exceeded errors from kubectl or controllers are client-side timeouts firing before a slow API server answers. Raise the client timeout, add retries with backoff in controllers, and check whether client-side rate limiting is throttling you before blaming the API server. Tune the client to the server's real latency, not to defaults meant for fast local clusters.

Not specified by the original author.

TL;DR

Run vault plugin register -sha256=${PLUGIN\SHA256} -command=${PLUGIN\COMMAND} auth ${PLUGIN\_MOUNT} 2. Run `eval vault auth enable...

Steps

1. The fix that worked in production was moving plugin initialization out of Factory into lazy one-time init. The reporter refactored the plugin so initialization happens once via sync.Once at first use, instead of inside the Factory function where a partially-initialized plugin could be published. They shipped this to production and it looked good. Note the remaining limitation from the thread: there is still no automatic recovery until the plugin process reloads.

When to use

You are seeing this: https://github.com/hashicorp/vault/issues/19192: Issue hashicorp/vault#19192 (closed, 40 comments): Hello again team, 👋 Describe the bug Vault is returning error: code = Canceled desc = context canceled" took=59.750225246s when trying to register an... Use this skill when you run into "Enabling Auth Plugin failing with context deadline exceeded".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Note the remaining limitation from the thread: there is still no automatic recovery until the plugin process reloads.

Sources

• https://github.com/hashicorp/vault/issues/19192:

Issue gh:hashicorp/vault#19192: the reporter asked: Issue hashicorp/vault#19192 (closed, 40 comments): Hello again team, 👋

Describe the bug Vault is returning error: code = Canceled desc = context canceled" took=59.750225246s when trying to register an external auth plugin. The debug is followed by the error 2023-02-15T00:58:57.417Z [ERROR] secrets.system.system_7f9e5cfe: error occurred during enable credential: path=$PLUGIN_PATH/ error="rpc error: code = Canceled desc = context canceled"

To Reproduce Steps to reproduce the behavior:

1. Run vault plugin register -sha256=${PLUGIN_SHA256} -command=${PLUGIN_COMMAND} auth ${PLUGIN_MOUNT}
2. Run eval vault auth enable... The verified answer was: The fix that worked in production was moving plugin initialization out of Factory into lazy one-time init. The reporter refactored the plugin so initialization happens once via sync.Once at first use, instead of inside the Factory` function where a partially-initialized plugin could be published. They shipped this to production and it looked good. Note the remaining limitation from the thread: there is still no automatic recovery until the plugin process reloads. Evidence thread: https://github.com/hashicorp/vault/issues/19192