## TL;DR
A 429 means a client is exceeding its request rate to the API server. Find the offending client by its user agent in the audit logs, then raise its QPS and burst settings or cut its request churn. Do not just retry harder; that makes it worse.

## Verbatim error
```text
Error from server (TooManyRequests): the server has received too many requests and asks the client to slow down
```

## Steps
1. Identify which client is being throttled from the audit log.
```bash
grep -i "429" [AUDIT-LOG-PATH]
```
Expected: log lines naming the user agent, for example a controller-manager, a custom operator, or a CI job. The user agent with the most 429s is your target. Use your cluster's log access method for the audit log path.

2. Check the current QPS and burst settings on in-cluster components.
```bash
ps aux | grep kube-controller-manager
```
Expected: flags like --kube-api-qps and --kube-api-burst with their values. Defaults are conservative (often 20 QPS); busy clusters need more.

3. Raise QPS and burst on the throttled component.
```bash
kubectl -n kube-system edit deployment [COMPONENT]
```
Expected: the component restarts with higher --kube-api-qps and --kube-api-burst values, and 429s for its user agent drop. Raise both together; burst absorbs spikes, QPS sets the sustained rate.

4. For a custom client-go application, raise the rate limiter in its kubeconfig-based client setup: increase the QPS and Burst fields on the rest config the app builds its client from, then redeploy.
```bash
kubectl rollout restart deployment/[APP] -n [NAMESPACE]
```
Expected: the app's 429 rate falls after the restart. If it does not, the app is generating too many requests and needs fewer watches, not a bigger limiter.

5. Cut request churn at the source: replace polling loops with watches, share informers across controllers in one process, and cache list calls.
```bash
kubectl top pods -n [NAMESPACE]
```
Expected: steady CPU and memory on the client pods. A client that was hammering the API usually shows high CPU from constant list calls; that should flatten after the change.

## When to use
- Controllers, operators, or CI jobs log TooManyRequests from the API server
- 429s spike during deployments, cronjob waves, or node churn
- Audit logs show one user agent dominating throttled requests

## When not to use
- The 429 comes from a cloud load balancer or ingress in front of the API; that is edge rate limiting, not client throttling
- Every client gets 429s at once; that suggests API server overload or etcd slowness, a server-side problem
- The client is a human running kubectl in a loop; fix the script, not the limiter

## Why it happens
client-go rate-limits itself with a token bucket (QPS for sustained rate, burst for spikes). When a client exhausts its bucket, its own limiter delays requests, and if the server-side priority and fairness queues are also saturated, the API server returns 429. Busy controllers with default limiter settings hit this first.

## Edge cases
- Raising QPS on every client just moves the bottleneck to the API server or etcd; raise selectively for the proven hot client.
- Flow control (PriorityLevelConfiguration) can 429 even well-behaved clients during overload; check kubectl get flowschemas during incidents.
- Operators that list all namespaces on every reconcile are the classic offender; scope them with namespace selectors first.
- 429s that persist after raising limits usually mean the API server itself is CPU-starved; check apiserver CPU and etcd latency.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_OyUgOB_lV9hlzqhSPPT9GA