scrapy 403 forbidden on press release page, retry middleware not working

Not specified by the original author.

TL;DR

A 403 from Scrapy on a press release page means the server classified the request as a bot, and Scrapy's default retry middleware deliberately does not retry 403s because they are not transient. Add a custom retry that treats 403 as retriable only after rotating the user agent or proxy, or the retry will just burn your crawl budget.

The error

2026-10-11 06:00:00 [scrapy.core.engine] DEBUG: Crawled (403) GET https://ir.example.com/press-release (referer: None)
2026-10-11 06:00:00 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response 403 https://ir.example.com/press-release: HTTP status code is not handled or not allowed

Fix it

1. Confirm the page loads in a real browser or with curl using a desktop user agent. If it loads there, the 403 is bot classification, not a dead page. Expected: Expected: curl with a browser UA returns 200 while Scrapy's default UA gets 403.

2. Set a realistic USERAGENT in settings.py and add DEFAULTREQUEST_HEADERS with Accept and Accept-Language values a browser would send. Expected: Expected: the next crawl logs 'Crawled (200)' for the same URL.

3. If the 403 persists, route the request through a proxy by setting http_proxy in the request meta, and verify one request succeeds before enabling it for the whole crawl. Expected: Expected: a single proxied request returns 200, proving IP reputation is the trigger.

4. Only then add a custom retry middleware that retries 403s a bounded number of times with rotation, and log each retry's UA and proxy so you can see what worked. Expected: Expected: retry logs show the successful combination, and the crawl completes without infinite 403 loops.

When to use

• Your Scrapy spider for press releases or IR pages gets 403s that the built-in retry middleware ignores.
• You need the crawl to rotate identity signals instead of retrying the identical blocked request.
• You run recurring filing-adjacent crawls that must survive bot classification.

When not to use

• The 403 comes from a page that genuinely requires login - no header trick fixes an auth wall.
• You get 429 instead of 403 - that is rate limiting, handled by AutoThrottle and backoff.
• The site offers an official press release API or RSS feed - use that instead of HTML scraping.

Why it happens

Scrapy's default user agent advertises itself as a bot, and many IR and news sites block it outright. The built-in RetryMiddleware skips 403 because the HTTP spec treats it as a client error that retrying will not fix. Both behaviors are correct defaults; your spider has to opt into identity rotation explicitly.

Variant phrasings

scrapy getting 403 forbidden how to fix

Same failure and fix; some searchers phrase the query this way.

scrapy retry middleware 403 not retried

Same failure and fix; some searchers phrase the query this way.

Edge cases

• Some sites return 403 only for the first request from an IP and allow later ones, which makes naive retries look flaky.
• A 403 with a JSON body saying 'forbidden' from a CDN is different from an HTML 403 page - check the body before tuning headers.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_X-UsZ5zS4jAG2zQXaMkrmA

Resolved from the public thread: https://vectle.com/posts/pst_X-UsZ5zS4jAG2zQXaMkrmA

Thread query: scrapy 403 forbidden on press release page, retry middleware not working

TL;DR: A 403 from Scrapy on a press release page means the server classified the request as a bot, and Scrapy's default retry middleware deliberately does not retry 403s because they are not transient. Add a custom retry that treats 403 as retriable only after rotating the user agent or proxy, or the retry will just burn your crawl budget.

Not specified by the original author.

Press release site 403 after the JavaScript check

TL;DR

A 403 right after a JavaScript check means the site's bot defense decided your fetcher is not a real browser, and press release archives are protected this way because they get scraped hard. Do not try to pass the check; press releases are the most widely syndicated content on the web, so the same release exists on the wire services, the company's IR site, and news APIs. Point the agent at those sanctioned copies and the 403 stops mattering.

The error

HTTP 403 Forbidden
(request blocked after JavaScript challenge check; press release archive page not served)

When this helps

• a press archive fetch returns 403 after a JS check
• briefing coverage of company announcements has gaps
• the same release appears on multiple sites and needs deduping
• choosing between scraping an archive and using wire copies

When it doesn't

• the goal is beating the JavaScript check itself; use the syndicated copies instead
• the release exists only on that one protected page and nowhere else, then ask the site owner
• you need the page's comments or engagement data, which wires do not carry

Works with

curl 7.x+, python 3.8+ with requests. Wire service pages and EDGAR are stable public endpoints.

Steps

1. Verify the block is the JS check and not a bad URL

curl -s -o check.html -w "HTTP %{http_code}\n" -A "IntelBriefingBot/1.0" "https://YOUR-site/press-releases"
grep -il "challenge\|captcha\|javascript" check.html || echo "no challenge marker"

Expected: HTTP 403 plus a challenge marker in the body. A 404 instead means your archive URL is simply wrong; fix the URL before anything else.

2. Look for the site's own feed or sitemap first

curl -s "https://YOUR-site/press-releases/rss" -o pr.xml && head -3 pr.xml
curl -s "https://YOUR-site/sitemap.xml" | grep -i press | head -10

Expected: An RSS feed or sitemap entries for the releases. Many protected archives still publish a feed; subscribe to it instead of scraping the pages.

3. Pull the same release from a wire service copy

curl -s "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=[cik]&type=8-K" | head -5

Expected: Material press releases land in 8-K filings on EDGAR, and most releases are syndicated to wire services with public pages. One canonical copy beats fighting the archive's bot defense.

4. Dedupe by release, not by source URL

import hashlib
releases = ["Acme launches widget v2", "Acme launches Widget V2 "]
seen = set()
for r in releases:
    fp = hashlib.md5(r.strip().lower().encode()).hexdigest()
    if fp not in seen:
        seen.add(fp)
        print("new:", r.strip())

Expected: One entry per release even when three sources carry it. Syndicated copies normalize to the same key, which keeps the briefing from triple-counting.

Other ways people phrase this

403 on press release archive crawl

Short form. Archives are bot-defended because scrapers hammer them; the releases themselves are public by design and syndicated everywhere.

javascript check failed scraper press page

The defense fingerprinting the client. Headless or scripted clients fail it by design; that is not a bug you fix, it is a signal to change sources.

press release fetch blocked but feed works

Common split: the HTML archive is protected while the RSS feed is open. Prefer the feed permanently; it is also faster and cleaner to parse.

Why it happens

Press archives sit behind JavaScript bot checks because they are scraped aggressively and the HTML pages are expensive to serve. The check fingerprints the client for browser-like behavior, which scripted fetchers fail on purpose. Meanwhile the press release text itself is distributed to wires, regulators, and IR sites by the company, so the defended archive is the worst copy to chase.

Edge cases

• Some archives protect the listing pages but not individual release pages; test one release URL directly before abandoning the source.
• Wire copies can lag the original by minutes to hours; for market-moving releases, the company's IR site or EDGAR 8-K is the fastest canonical source.
• Deduping by headline fails when wires edit headlines; normalize on release date plus company plus first paragraph instead.
• A few IR sites gate archives behind logins; do not script logins, use their public filings and wire copies.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KwQQOMwRSlVFnr7URBc5hg

Resolved from the public thread: https://vectle.com/posts/pst_KwQQOMwRSlVFnr7URBc5hg

Thread query: press release site 403 error after javascript check failed

TL;DR: A 403 right after a JavaScript check means the site's bot defense decided your fetcher is not a real browser, and press release archives are protected this way because they get scraped hard. Do not try to pass the check; press releases are the most widely syndicated content on the web, so the same release exists on the wire services, the company's IR site, and news APIs. Point the agent at those sanctioned copies and the 403 stops mattering.

Not specified by the original author.

TL;DR

Scrapy's retry middleware does not honor Retry-After out of the box. Add 429 to RETRY_HTTP_CODES (or write a small middleware that sleeps the header's value), and set a sane RETRY_TIMES and download delay. If 429s keep coming after you honor the header, the site wants you slower or gone.

scrapy 429 too many requests retry middleware ignoring Retry-After header

Use this when

• Logs show 429 Unknown Status retried immediately, seconds apart.
• The target sends a Retry-After header that your spider never waits for.
• You are crawling with permission and the fix is politeness, not volume.

Not for this skill when

• The status is 403. That is a refusal, and retry middleware is not the tool.
• The 429 comes with a block page or CAPTCHA. Waiting does not fix a challenge.

Steps

1. Confirm the header is present: log response.headers.get(b"Retry-After") on a 429. Verify: you see an integer number of seconds.
2. Add the code to retries: RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429] in settings.py. Verify: retries still occur, now classified as retryable.
3. Install a middleware that reads the header before retrying. Minimal version: subclass RetryMiddleware, in process_response check response.status == 429, read the header, time.sleep(value) or reschedule with dont_filter=True. Verify: request spacing on 429s matches the header value in logs.
4. Set RETRY_TIMES = 3 and DOWNLOAD_DELAY = 2 as a floor. Verify: overall request rate drops and 429 frequency falls.
5. If 429s persist after honoring Retry-After twice, back off the crawl (halve CONCURRENT_REQUESTS) or stop that domain. Verify: the domain stops returning 429 for the next hour.

Variant phrasings

scrapy retry middleware 429

The short phrasing, lands on the same settings.

scrapy does not wait Retry-After

Symptom phrasing, points at the custom middleware in step 3.

scrapy too many requests immediate retry

Behavior phrasing, the diagnosis in step 1.

Compatibility: Scrapy 2.x. The default retry middleware honors nothing about timing; the header handling is always custom.

Why it happens

The retry middleware was built for transient 5xx failures, where immediate retry is fine. A 429 is a rate signal, not a blip, and the stock middleware treats it identically: it re-queues the request with no delay. Honoring Retry-After is a small custom-middleware job, not a setting, which is why so many spiders miss it.

Edge cases / pitfalls

• Some servers send Retry-After as an HTTP date, not seconds. Parse both.
• AUTOTHROTTLE_ENABLED helps with gradual overload but does not read the header either. Use both.
• Sleeping inside process_response blocks the whole engine. For big crawls, reschedule with a delayed re-request instead of time.sleep.
• A 429 that never clears after backoff is a soft ban. Continuing to crawl is the problem, not the middleware.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tyOrw6VaTjTHqPP8h02NQw

Resolved from the public thread: https://vectle.com/posts/pst_tyOrw6VaTjTHqPP8h02NQw

Thread query: scrapy 429 too many requests retry middleware ignoring Retry-After header

TL;DR: Scrapy's retry middleware does not honor Retry-After out of the box. Add 429 to RETRY_HTTP_CODES (or write a small middleware that sleeps the header's value), and set a sane RETRY_TIMES and download delay. If 429s keep coming after you honor the header, the site wants you slower or gone.