scrapy 403 forbidden on press release page, retry middleware not working

TL;DR

A 403 from Scrapy on a press release page means the server classified the request as a bot, and Scrapy's default retry middleware deliberately does not retry 403s because they are not transient. Add a custom retry that treats 403 as retriable only after rotating the user agent or proxy, or the retry will just burn your crawl budget.

The error

2026-10-11 06:00:00 [scrapy.core.engine] DEBUG: Crawled (403) GET https://ir.example.com/press-release (referer: None)
2026-10-11 06:00:00 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response 403 https://ir.example.com/press-release: HTTP status code is not handled or not allowed

Fix it

1. Confirm the page loads in a real browser or with curl using a desktop user agent. If it loads there, the 403 is bot classification, not a dead page. Expected: Expected: curl with a browser UA returns 200 while Scrapy's default UA gets 403.

2. Set a realistic USERAGENT in settings.py and add DEFAULTREQUEST_HEADERS with Accept and Accept-Language values a browser would send. Expected: Expected: the next crawl logs 'Crawled (200)' for the same URL.

3. If the 403 persists, route the request through a proxy by setting http_proxy in the request meta, and verify one request succeeds before enabling it for the whole crawl. Expected: Expected: a single proxied request returns 200, proving IP reputation is the trigger.

4. Only then add a custom retry middleware that retries 403s a bounded number of times with rotation, and log each retry's UA and proxy so you can see what worked. Expected: Expected: retry logs show the successful combination, and the crawl completes without infinite 403 loops.

When to use

• Your Scrapy spider for press releases or IR pages gets 403s that the built-in retry middleware ignores.
• You need the crawl to rotate identity signals instead of retrying the identical blocked request.
• You run recurring filing-adjacent crawls that must survive bot classification.

When not to use

• The 403 comes from a page that genuinely requires login - no header trick fixes an auth wall.
• You get 429 instead of 403 - that is rate limiting, handled by AutoThrottle and backoff.
• The site offers an official press release API or RSS feed - use that instead of HTML scraping.

Why it happens

Scrapy's default user agent advertises itself as a bot, and many IR and news sites block it outright. The built-in RetryMiddleware skips 403 because the HTTP spec treats it as a client error that retrying will not fix. Both behaviors are correct defaults; your spider has to opt into identity rotation explicitly.

Variant phrasings

scrapy getting 403 forbidden how to fix

Same failure and fix; some searchers phrase the query this way.

scrapy retry middleware 403 not retried

Same failure and fix; some searchers phrase the query this way.

Edge cases

• Some sites return 403 only for the first request from an IP and allow later ones, which makes naive retries look flaky.
• A 403 with a JSON body saying 'forbidden' from a CDN is different from an HTML 403 page - check the body before tuning headers.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_X-UsZ5zS4jAG2zQXaMkrmA

Resolved from the public thread: https://vectle.com/posts/pst_X-UsZ5zS4jAG2zQXaMkrmA

Thread query: scrapy 403 forbidden on press release page, retry middleware not working

TL;DR: A 403 from Scrapy on a press release page means the server classified the request as a bot, and Scrapy's default retry middleware deliberately does not retry 403s because they are not transient. Add a custom retry that treats 403 as retriable only after rotating the user agent or proxy, or the retry will just burn your crawl budget.