AWS "rate exceeded" API errors: backoff strategy

TL;DR

AWS "rate exceeded" means you are calling an API faster than your quota, and the fix is client-side: exponential backoff with jitter, plus batching calls that do not need to be individual. Retrying immediately makes it worse; the throttle lasts longer the harder you hammer it. Find the throttled calls in CloudTrail, add backoff, and check whether a quota increase is actually warranted.

Error / query

AWS "rate exceeded" API errors: backoff strategy

Use this skill when

• Logs show RateExceeded or ThrottlingException from AWS SDK calls
• A deploy or autoscaling event suddenly starts failing with API errors
• CloudTrail shows a spike in throttled API calls
• You need a backoff pattern to put in code or in a script

Not for this skill when

• The error is AccessDenied or InvalidParameter; those are not throttling
• You are hitting a hard service limit (like max VPCs); request a limit increase instead
• The throttling is on someone else's API; backoff still helps but you cannot raise their quota

Steps

Step 1: Find which API calls are being throttled

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DescribeInstances --max-items 5 2>&1 | head -20

Expected: recent events; check your application logs for the exact "Rate exceeded" error text to identify the call.

Step 2: Check the current throttle rate in your app logs

journalctl -u myservice --since "1 hour ago" | grep -ci 'rate exceeded\|throttl'

Expected: a count; if it is climbing, the retry storm is feeding itself and you need backoff now.

Step 3: Add exponential backoff with jitter to the retry loop

printf 'import random, time\ndef aws_call_with_backoff(fn, tries=5):\n    for i in range(tries):\n        try:\n            return fn()\n        except Exception:\n            time.sleep(min(2 ** i + random.uniform(0, 1), 20))\n    return fn()\n' > /tmp/backoff.py && cat /tmp/backoff.py

Expected: the helper prints back; sleeps grow 1s, 2s, 4s, 8s plus random jitter, capped at 20s.

Step 4: Verify the SDK is using the standard retry config

python3 -c "import botocore.config; c=botocore.config.Config(retries={'max_attempts': 5, 'mode': 'adaptive'}); print(c.retries)"

Expected: the adaptive retry config prints; adaptive mode is the one that actually backs off on throttling.

Step 5: Check whether the quota itself needs raising

aws service-quotas list-service-quotas --service-code ec2 --query 'Quotas[?contains(QuotaName, `Rate`)].{Name:QuotaName,Value:Value}' --output table 2>&1 | head -15

Expected: the API rate quotas; if you are legitimately near the quota with good backoff, file an increase request.

Variant phrasings

"ThrottlingException in boto3"

Same fix: adaptive retry mode plus jittered backoff; boto3's default legacy mode retries too aggressively.

"AWS API rate limits during Terraform apply"

Terraform hammers Describe calls; set parallelism lower and space applies out, or the backoff in the provider does the waiting for you.

"How to add jitter to retries"

Add a random 0-1 second (or 0-100% of the delay) to every sleep; without jitter, all your clients retry in lockstep and re-throttle each other.

Why it happens

AWS throttles per API per region to protect the control plane; SDKs retry by default, and naive immediate retries turn one throttle into a sustained storm. Backoff with jitter spreads the retries out so the quota can recover.

Edge cases and pitfalls

• Retrying a throttled write (Create, Delete) is safe; retrying is only dangerous for non-idempotent calls, which AWS APIs mostly are not.
• Some throttles come from a shared quota across all your tooling; one noisy script can throttle your deploys.
• Do not "fix" throttling by requesting a quota increase for a retry storm; fix the client first, then measure.
• Cross-account or cross-region calls have separate quotas; the throttle may be in the assumed role's account, not yours.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mufS8GBwRVhPOa3MwS3bhQ