Kubernetes API 429 Too Many Requests: client-side throttling fix

Not specified by the original author.

TL;DR

A 429 means a client is exceeding its request rate to the API server. Find the offending client by its user agent in the audit logs, then raise its QPS and burst settings or cut its request churn. Do not just retry harder; that makes it worse.

Verbatim error

Error from server (TooManyRequests): the server has received too many requests and asks the client to slow down

Steps

1. Identify which client is being throttled from the audit log.

grep -i "429" [AUDIT-LOG-PATH]

Expected: log lines naming the user agent, for example a controller-manager, a custom operator, or a CI job. The user agent with the most 429s is your target. Use your cluster's log access method for the audit log path.

2. Check the current QPS and burst settings on in-cluster components.

ps aux | grep kube-controller-manager

Expected: flags like --kube-api-qps and --kube-api-burst with their values. Defaults are conservative (often 20 QPS); busy clusters need more.

3. Raise QPS and burst on the throttled component.

kubectl -n kube-system edit deployment [COMPONENT]

Expected: the component restarts with higher --kube-api-qps and --kube-api-burst values, and 429s for its user agent drop. Raise both together; burst absorbs spikes, QPS sets the sustained rate.

4. For a custom client-go application, raise the rate limiter in its kubeconfig-based client setup: increase the QPS and Burst fields on the rest config the app builds its client from, then redeploy.

kubectl rollout restart deployment/[APP] -n [NAMESPACE]

Expected: the app's 429 rate falls after the restart. If it does not, the app is generating too many requests and needs fewer watches, not a bigger limiter.

5. Cut request churn at the source: replace polling loops with watches, share informers across controllers in one process, and cache list calls.

kubectl top pods -n [NAMESPACE]

Expected: steady CPU and memory on the client pods. A client that was hammering the API usually shows high CPU from constant list calls; that should flatten after the change.

When to use

• Controllers, operators, or CI jobs log TooManyRequests from the API server
• 429s spike during deployments, cronjob waves, or node churn
• Audit logs show one user agent dominating throttled requests

When not to use

• The 429 comes from a cloud load balancer or ingress in front of the API; that is edge rate limiting, not client throttling
• Every client gets 429s at once; that suggests API server overload or etcd slowness, a server-side problem
• The client is a human running kubectl in a loop; fix the script, not the limiter

Why it happens

client-go rate-limits itself with a token bucket (QPS for sustained rate, burst for spikes). When a client exhausts its bucket, its own limiter delays requests, and if the server-side priority and fairness queues are also saturated, the API server returns 429. Busy controllers with default limiter settings hit this first.

Edge cases

• Raising QPS on every client just moves the bottleneck to the API server or etcd; raise selectively for the proven hot client.
• Flow control (PriorityLevelConfiguration) can 429 even well-behaved clients during overload; check kubectl get flowschemas during incidents.
• Operators that list all namespaces on every reconcile are the classic offender; scope them with namespace selectors first.
• 429s that persist after raising limits usually mean the API server itself is CPU-starved; check apiserver CPU and etcd latency.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstOyUgOBlV9hlzqhSPPT9GA

Resolved from the public thread: https://vectle.com/posts/pstOyUgOBlV9hlzqhSPPT9GA

Thread query: Kubernetes API "429 Too Many Requests": client-side throttling fix

TL;DR: A 429 means a client is exceeding its request rate to the API server. Find the offending client by its user agent in the audit logs, then raise its QPS and burst settings or cut its request churn. Do not just retry harder; that makes it worse.

Not specified by the original author.

TL;DR

Most context deadline exceeded errors from kubectl or controllers are client-side timeouts firing before a slow API server answers. Raise the client timeout, add retries with backoff in controllers, and check whether client-side rate limiting is throttling you before blaming the API server. Tune the client to the server's real latency, not to defaults meant for fast local clusters.

Error / query

how to tune Kubernetes API client timeouts and retries

Use this skill when

• kubectl or controllers report context deadline exceeded
• API calls flap or time out under load
• Automation is slow and client-side throttling is suspected
• Controllers log rate-limiter waits

Not for this skill when

• The API server itself is slow or overloaded (server-side tuning)
• etcd is the bottleneck (different diagnosis)
• Requests are denied by RBAC (authorization, not timeouts)

Steps

Step 1: Measure whether the server or the client is slow

time kubectl get pods -n [namespace] --request-timeout=60s

Expected: the wall time tells you the server's real latency. If --request-timeout=60s succeeds where the default (30s) failed, the client timeout was the problem, not the server.

Step 2: Set a sane request timeout for interactive use

kubectl get pods -n [namespace] --request-timeout=2m

Expected: long list/watch calls complete instead of timing out. For scripts, export it per-command rather than relying on the default; large list calls against big clusters legitimately take over 30s.

Step 3: Tune client-go rate limiting in controllers

config.QPS = 50
config.Burst = 100

Expected: in controller code, raising QPS/Burst from the client-go defaults (5/10) stops the client from self-throttling. Watch controller logs for Waited for messages from the rate limiter; their disappearance confirms the fix.

Step 4: Add retries with backoff around API calls

for i in 1 2 3 4 5; do kubectl get [resource] [name] -n [namespace] && break || sleep $((2 ** i)); done

Expected: transient timeouts succeed on retry with exponential backoff. For controllers, use the client-go retry utilities rather than failing the reconcile loop on the first timeout.

Variant phrasings

"kubectl context deadline exceeded"

Step 1 first: raise --request-timeout and see if the call succeeds. If it does, it was the client.

"kubernetes client throttling"

Check controller logs for rate-limiter waits, then raise QPS/Burst as in step 3.

Why it happens

Client defaults assume a fast, nearby API server: 30s request timeouts and aggressive client-side rate limits. Against large, loaded, or distant clusters, legitimate calls exceed those defaults, and the client reports a timeout that looks like a server failure. The fix is matching client patience to server reality.

Edge cases and pitfalls

• Raising timeouts masks a genuinely sick API server; if p99 latency keeps climbing, tune the server instead of the client.
• Watch calls with no timeout can hang forever on network partitions; always bound watches, even generous ones.
• Very high QPS from many controllers can DDoS a small API server; raise limits gradually and watch API server latency.
• --request-timeout=0 means no timeout, which is dangerous in scripts; always set an explicit bound.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jf7Qv9jBV9iWwtT-gg-2fA

Resolved from the public thread: https://vectle.com/posts/pst_jf7Qv9jBV9iWwtT-gg-2fA

Thread query: how to tune Kubernetes API client timeouts and retries

TL;DR: Most context deadline exceeded errors from kubectl or controllers are client-side timeouts firing before a slow API server answers. Raise the client timeout, add retries with backoff in controllers, and check whether client-side rate limiting is throttling you before blaming the API server. Tune the client to the server's real latency, not to defaults meant for fast local clusters.

Not specified by the original author.

TL;DR

Add exponential backoff with jitter to the client. What changed in the last hour?

Steps

1. Symptom (exact): API responses 429 Too Many Requests, error body says too many requests. Starts suddenly, often right after a deploy or a new script.
2. Likely causes: (a) a new polling script in a tight loop, (b) Terraform/CI fanning out hundreds of calls, (c) a retry storm: the first 429s triggered immediate retries that extended the throttle.
3. What changed in the last hour? New cron, new exporter, new Terraform run. The 429 start time usually matches a deploy.
4. Which endpoint 429s? If it is one endpoint from one client, that client is the offender. If everything 429s, something is hammering broadly.
5. Check for retry-without-backoff in the client code. Immediate retry on 429 is a throttle amplifier.
6. Stop the offender first (kill the loop), then wait out the throttle honoring Retry-After.
7. Add exponential backoff with jitter to the client.
8. Cut volume: cache, batch, widen intervals. A dashboard polled every 10s from CI should be a monitor, not a script.
9. Verify: 429s stop for 15 minutes of normal operation, then re-enable the fixed client and watch. If they return, the volume cut was not enough: measure calls per minute per endpoint and budget accordingly.

When to use

You are seeing this: Symptom (exact): API responses 429 Too Many Requests, error body says too many requests. Use this skill when you run into "Diag: 429 rate limited, find the hammering client, backoff, cut volume".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

The API reference returns 429 Too many requests as a standard response (documented across endpoints, e.g. the Monitors API response codes). Verified against live docs 2026-09-26.