how to tune Kubernetes API client timeouts and retries

TL;DR

Most context deadline exceeded errors from kubectl or controllers are client-side timeouts firing before a slow API server answers. Raise the client timeout, add retries with backoff in controllers, and check whether client-side rate limiting is throttling you before blaming the API server. Tune the client to the server's real latency, not to defaults meant for fast local clusters.

Error / query

how to tune Kubernetes API client timeouts and retries

Use this skill when

• kubectl or controllers report context deadline exceeded
• API calls flap or time out under load
• Automation is slow and client-side throttling is suspected
• Controllers log rate-limiter waits

Not for this skill when

• The API server itself is slow or overloaded (server-side tuning)
• etcd is the bottleneck (different diagnosis)
• Requests are denied by RBAC (authorization, not timeouts)

Steps

Step 1: Measure whether the server or the client is slow

time kubectl get pods -n [namespace] --request-timeout=60s

Expected: the wall time tells you the server's real latency. If --request-timeout=60s succeeds where the default (30s) failed, the client timeout was the problem, not the server.

Step 2: Set a sane request timeout for interactive use

kubectl get pods -n [namespace] --request-timeout=2m

Expected: long list/watch calls complete instead of timing out. For scripts, export it per-command rather than relying on the default; large list calls against big clusters legitimately take over 30s.

Step 3: Tune client-go rate limiting in controllers

config.QPS = 50
config.Burst = 100

Expected: in controller code, raising QPS/Burst from the client-go defaults (5/10) stops the client from self-throttling. Watch controller logs for Waited for messages from the rate limiter; their disappearance confirms the fix.

Step 4: Add retries with backoff around API calls

for i in 1 2 3 4 5; do kubectl get [resource] [name] -n [namespace] && break || sleep $((2 ** i)); done

Expected: transient timeouts succeed on retry with exponential backoff. For controllers, use the client-go retry utilities rather than failing the reconcile loop on the first timeout.

Variant phrasings

"kubectl context deadline exceeded"

Step 1 first: raise --request-timeout and see if the call succeeds. If it does, it was the client.

"kubernetes client throttling"

Check controller logs for rate-limiter waits, then raise QPS/Burst as in step 3.

Why it happens

Client defaults assume a fast, nearby API server: 30s request timeouts and aggressive client-side rate limits. Against large, loaded, or distant clusters, legitimate calls exceed those defaults, and the client reports a timeout that looks like a server failure. The fix is matching client patience to server reality.

Edge cases and pitfalls

• Raising timeouts masks a genuinely sick API server; if p99 latency keeps climbing, tune the server instead of the client.
• Watch calls with no timeout can hang forever on network partitions; always bound watches, even generous ones.
• Very high QPS from many controllers can DDoS a small API server; raise limits gradually and watch API server latency.
• --request-timeout=0 means no timeout, which is dangerous in scripts; always set an explicit bound.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jf7Qv9jBV9iWwtT-gg-2fA