Kubernetes API "429 Too Many Requests": client-side throttling fix

TL;DR

A 429 means a client is exceeding its request rate to the API server. Find the offending client by its user agent in the audit logs, then raise its QPS and burst settings or cut its request churn. Do not just retry harder; that makes it worse.

Verbatim error

Error from server (TooManyRequests): the server has received too many requests and asks the client to slow down

Steps

1. Identify which client is being throttled from the audit log.

grep -i "429" [AUDIT-LOG-PATH]

Expected: log lines naming the user agent, for example a controller-manager, a custom operator, or a CI job. The user agent with the most 429s is your target. Use your cluster's log access method for the audit log path.

2. Check the current QPS and burst settings on in-cluster components.

ps aux | grep kube-controller-manager

Expected: flags like --kube-api-qps and --kube-api-burst with their values. Defaults are conservative (often 20 QPS); busy clusters need more.

3. Raise QPS and burst on the throttled component.

kubectl -n kube-system edit deployment [COMPONENT]

Expected: the component restarts with higher --kube-api-qps and --kube-api-burst values, and 429s for its user agent drop. Raise both together; burst absorbs spikes, QPS sets the sustained rate.

4. For a custom client-go application, raise the rate limiter in its kubeconfig-based client setup: increase the QPS and Burst fields on the rest config the app builds its client from, then redeploy.

kubectl rollout restart deployment/[APP] -n [NAMESPACE]

Expected: the app's 429 rate falls after the restart. If it does not, the app is generating too many requests and needs fewer watches, not a bigger limiter.

5. Cut request churn at the source: replace polling loops with watches, share informers across controllers in one process, and cache list calls.

kubectl top pods -n [NAMESPACE]

Expected: steady CPU and memory on the client pods. A client that was hammering the API usually shows high CPU from constant list calls; that should flatten after the change.

When to use

• Controllers, operators, or CI jobs log TooManyRequests from the API server
• 429s spike during deployments, cronjob waves, or node churn
• Audit logs show one user agent dominating throttled requests

When not to use

• The 429 comes from a cloud load balancer or ingress in front of the API; that is edge rate limiting, not client throttling
• Every client gets 429s at once; that suggests API server overload or etcd slowness, a server-side problem
• The client is a human running kubectl in a loop; fix the script, not the limiter

Why it happens

client-go rate-limits itself with a token bucket (QPS for sustained rate, burst for spikes). When a client exhausts its bucket, its own limiter delays requests, and if the server-side priority and fairness queues are also saturated, the API server returns 429. Busy controllers with default limiter settings hit this first.

Edge cases

• Raising QPS on every client just moves the bottleneck to the API server or etcd; raise selectively for the proven hot client.
• Flow control (PriorityLevelConfiguration) can 429 even well-behaved clients during overload; check kubectl get flowschemas during incidents.
• Operators that list all namespaces on every reconcile are the classic offender; scope them with namespace selectors first.
• 429s that persist after raising limits usually mean the API server itself is CPU-starved; check apiserver CPU and etcd latency.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstOyUgOBlV9hlzqhSPPT9GA