Diag: 429 rate limited, find the hammering client, backoff, cut volume

Not specified by the original author.

Diag: your account is rate limited (429s)

Symptom (exact): API responses 429 Too Many Requests, error body says too many requests. Starts suddenly, often right after a deploy or a new script.

Likely causes: (a) a new polling script in a tight loop, (b) Terraform/CI fanning out hundreds of calls, (c) a retry storm: the first 429s triggered immediate retries that extended the throttle.

Confirm:

1. What changed in the last hour? New cron, new exporter, new Terraform run. The 429 start time usually matches a deploy.
2. Which endpoint 429s? If it is one endpoint from one client, that client is the offender. If everything 429s, something is hammering broadly.
3. Check for retry-without-backoff in the client code. Immediate retry on 429 is a throttle amplifier.

Fix:

1. Stop the offender first (kill the loop), then wait out the throttle honoring Retry-After.
2. Add exponential backoff with jitter to the client.
3. Cut volume: cache, batch, widen intervals. A dashboard polled every 10s from CI should be a monitor, not a script.

Verify: 429s stop for 15 minutes of normal operation, then re-enable the fixed client and watch. If they return, the volume cut was not enough: measure calls per minute per endpoint and budget accordingly.

The API reference returns 429 Too many requests as a standard response (documented across endpoints, e.g. the Monitors API response codes). Verified against live docs 2026-09-26.