reddit api 429 too many requests search, scrape failed

Reddit API 429 too many requests on search

TL;DR

Reddit's API 429s search calls when the agent exceeds the rate limit, roughly 100 queries per minute for OAuth clients, and Reddit enforces it strictly. The fix is OAuth with a descriptive user agent, polite pacing well under the limit, and honoring the retry hints. Do not scrape old.reddit or the JSON endpoints to dodge the limit; that violates the terms and earns IP bans.

The error

HTTP 429 Too Many Requests
(reddit search API throttled; x-ratelimit-reset header present)

When this helps

• Reddit API search calls return 429
• a social listening agent throttles on Reddit
• setting up Reddit API access for an agent
• designing polite Reddit polling

When it doesn't

• you want to scrape Reddit's web JSON to avoid limits; that violates the terms
• the error is 401; that is OAuth, not rate
• you need firehose volume; the API is not built for that, use a licensed feed

Works with

Reddit API with OAuth as of 2026; about 100 queries per minute for authenticated clients.

Steps

1. Use OAuth with a descriptive user agent

import requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
r = s.get("https://oauth.reddit.com/r/[subreddit]/search", params={"q": "[topic]", "restrict_sr": "on"}, timeout=20)
print(r.status_code, r.headers.get("x-ratelimit-remaining"))

Expected: A 200 with ratelimit headers. OAuth plus a real user agent gets the documented limits; anonymous or default agents get throttled harder.

2. Honor the ratelimit headers on every response

import requests, time
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
r = s.get("https://oauth.reddit.com/r/[subreddit]/new", timeout=20)
rem = float(r.headers.get("x-ratelimit-remaining", 1))
print("remaining:", rem)
if rem in (1, 2, 3, 4):
    time.sleep(10)

Expected: Proactive slowing near the limit. Reading the headers beats discovering the 429 after the fact.

3. Back off on 429 using the reset hint

import requests, time
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
r = s.get("https://oauth.reddit.com/search", params={"q": "[topic]"}, timeout=20)
if r.status_code == 429:
    wait = int(float(r.headers.get("x-ratelimit-reset", 60)))
    print("throttled, waiting", wait)
    time.sleep(wait)

Expected: A successful retry after the indicated wait. The reset header tells you exactly how long.

4. Cache search results by query and subreddit

import hashlib
q = "[topic]:[subreddit]"
fp = hashlib.md5(q.encode()).hexdigest()
print("cache:", "cache/reddit_" + fp + ".json")
print("search results change slowly; cache for hours")

Expected: A cache path. Repeat searches for the same topic are the most common quota waste.

Other ways people phrase this

reddit api 429 too many requests

The standard throttle. OAuth, user agent, header-driven pacing.

reddit search rate limit scraper

Scraping to dodge the limit is what gets IPs banned. Use the API politely.

x-ratelimit-reset reddit

The header with the wait time. Honor it instead of guessing.

Why it happens

Reddit rate-limits its API to protect the service, with documented limits for OAuth clients and much harsher throttling for anonymous or bot-default traffic. Search is one of the more expensive endpoints. The 429 with ratelimit headers is the system working; clients are expected to read the headers and pace themselves.

Edge cases

• The old JSON endpoints without OAuth are throttled aggressively; always use oauth.reddit.com.
• Pushshift and similar archives are third-party; verify their coverage before depending on them.
• Comment and submission streams have separate limits from search.
• A descriptive user agent is required by the API rules, not optional.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_BWGswfpqiRbklqgXZa4FzA