langchain "RateLimitError: exceeded quota" with exponential backoff
Handles LLM provider rate-limit errors in LangChain with exponential backoff. Use when RateLimitError or 429s hit under concurrency, when retries need jitter to avoid thundering herds, and when you must tell a rate limit apart from an exhausted billing quota. Not for invalid API keys or authentication errors.
TL;DR
Wrap the LLM call in a retry with exponential backoff and jitter, and cut concurrency. Backoff fixes "slow down" rate limits; it cannot fix a billing quota that is actually exhausted, so check which one you have first.
Error
openai.RateLimitError: Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-4o...', 'type': 'rate_limit_error'}}Steps
- Confirm it is a rate limit and not an empty billing quota. Rate-limit messages say "rate limit reached" or "too many requests"; quota messages mention billing, credit balance, or "exceeded your current quota" with a billing link.
Expected output: you know which problem you have. Only the rate-limit kind is fixed by backoff.
- Wrap the call with exponential backoff using tenacity:
from tenacity import retry, wait_exponential, stop_after_attempt, retry_if_exception_type
from openai import RateLimitError
@retry(
wait=wait_exponential(multiplier=1, min=2, max=60),
stop=stop_after_attempt(6),
retry=retry_if_exception_type(RateLimitError),
reraise=True,
)
def call_llm(prompt):
return llm.invoke(prompt)Expected output: transient 429s are retried with waits of roughly 2, 4, 8, 16, 32, 60 seconds instead of crashing the chain.
- Add jitter so parallel workers do not retry in lockstep.
wait_exponentialalready randomizes, or combine explicitly:
from tenacity import wait_random
wait = wait_exponential(multiplier=1, max=60) + wait_random(0, 2)Expected output: retries spread out instead of hitting the provider as one synchronized wave.
- Reduce concurrency at the source. Cap parallel chains:
chain = prompt | llm
results = chain.batch(inputs, config={"max_concurrency": 3})Expected output: fewer simultaneous requests, so the backoff rarely triggers at all.
When to use
- RateLimitError or HTTP 429 under parallel or batch workloads
- Retries hammer the provider in sync and keep failing
- You need to distinguish throttling from a dead billing quota
When not to use
- Invalid API key or authentication errors (401s never heal with waiting)
- The quota is genuinely exhausted (upgrade the plan or wait for the billing reset)
- A single sequential call fails (thats not a rate problem, read the error)
Variant phrasings
anthropic RateLimitError with langchain
Same pattern; catch the Anthropic rate-limit exception type and back off identically.
429s from Azure OpenAI deployments
Azure throttles per deployment. Back off the same way, and consider spreading load across two deployments.
Why it happens
Providers throttle requests-per-minute and tokens-per-minute. A burst of parallel LangChain calls trips the throttle even when total usage is modest; the API is asking you to slow down, not go away.
Edge cases
- Only retry on 429/rate-limit types; retrying 401s or 400s wastes time and can lock the key.
- Honor a Retry-After header when the provider sends one; it beats your computed wait.
- In long batch jobs, combine backoff with a global rate limiter so the job degrades gracefully instead of stalling on retries.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_Ibin6dM40953EIE6nIuWpA