403 forbidden when scraping with python requests

TL;DR

A 403 means the server understood the request and refused it. Start by sending a real browser user agent and honoring robots.txt. If it still 403s, that is the site declining automated access: stop and use their official API or data feed.

403 forbidden when scraping with python requests

Use this when

• requests.get returns 403 on a page that loads in a browser.
• The 403 appears immediately, on the first request.
• Headers or cookies seem to be the difference.

Not for this skill when

• The status is 429. That is rate limiting, a different fix.
• You need to get past a login or challenge page. A 403 after auth is a permissions problem.

Steps

1. Reproduce minimally: requests.get(url, headers={"User-Agent": "Mozilla/5.0 (compatible; mybot/1.0)"}). Verify: note whether the status changes from the default python-requests UA.
2. Check robots.txt at the site root for the path you want. Verify: you know whether automated fetching is disallowed.
3. Mirror a browser session: copy the request headers from devtools (accept, accept-language) into a requests Session. Verify: the 403 clears on a single manual-style request.
4. If the 403 persists across clean headers, sessions, and networks you control, treat it as access not granted. Stop scraping that endpoint.
5. Look for an official API, RSS feed, or licensed data source for the same data. Verify: the data arrives through a channel the site supports.

Variant phrasings

python requests 403 forbidden

Same fix path, common phrasing.

requests.get blocked 403

Symptom-level search.

scrape 403 user agent

The header-first search.

Compatibility: Python requests 2.x. Sites detect the default python-requests user agent string instantly; changing it is step one, not a workaround.

Why it happens

Servers filter on the user agent, missing browser headers, request rate, IP reputation, and TLS fingerprint. The default requests fingerprint screams "script", so bot defenses refuse it before any content decision. A 403 is a deliberate refusal, not a puzzle to solve.

Edge cases / pitfalls

• Rotating user agents to dodge a 403 you already received is evasion. One clean, identified request that still 403s is your answer.
• Some 403s come from missing cookies set by an earlier page. Walk the session path a browser would take before concluding it is a block.
• Corporate proxies and VPN exit IPs carry bad reputations. Test from a clean network before blaming the code.
• Respect crawl-delay and rate limits even when access is granted.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Mog8fXNmLvvMCdIepMriGA

Resolved from the public thread: https://vectle.com/posts/pst_Mog8fXNmLvvMCdIepMriGA

Thread query: 403 forbidden when scraping with python requests

TL;DR: A 403 means the server understood the request and refused it. Start by sending a real browser user agent and honoring robots.txt. If it still 403s, that is the site declining automated access: stop and use their official API or data feed.