## TL;DR
Almost always the fix is to make sure you are calling can_fetch with the same User-agent name you used in read(). If the agent string differs, robotparser treats it as a different agent and returns False. Also fetch robots.txt over https when the target redirects to https, since the parser reads the rules for the URL you passed.

```text
urllib.robotparser returns False for allowed path
```

## Use this when
- can_fetch("MyBot", some_url) returns False but the path looks allowed in robots.txt
- You recently changed the User-agent string or the read() URL scheme
- Rules work in other parsers (reppy, protector) but not urllib

## Not for this skill when
- The site owner actually disallows the path (this skill does not help evade that)
- You need per-request handling of robots rules in Scrapy (use Scrapy's ROBOTSTXT_OBEY settings instead)
- The block is a login wall, CAPTCHA, or 403 from a WAF rather than robots.txt

## Steps
1. Print what the parser actually parsed: call rp.site_maps() or dump the rules with rp.default_entry and rp.entries, and confirm the path you test is under the host you read. Expected: you see the rules matching the site you intend to crawl.
2. Use the exact same agent string everywhere: rp = urllib.robotparser.RobotFileParser(); rp.set_url(url); rp.read(); rp.can_fetch("MyBot", target) where "MyBot" also matches the User-agent token in robots.txt. Expected: the agent token matches a rule block; no default-deny applies.
3. Match the scheme: if the site redirects http to https, call set_url with the https robots.txt URL and re-run can_fetch. Expected: can_fetch reflects the canonical site's rules, not a redirect chain the parser never followed.
4. Clear stale rules: call rp.set_url(url) and rp.read() again right before can_fetch in long-lived processes; cached rules can predate an edited robots.txt. Expected: rules match the current file.
5. Verify with a second parser (protector or reppy) on the same file to rule out urllib quirks with wildcards. Expected: both parsers agree, or you have evidence urllib's glob handling is the outlier.

### Alternate for step 2 (wildcard agent)
Test can_fetch("*", target) and can_fetch("MyBot", target) side by side. Expected: "*" matching proves the rule block exists; the difference names the agent-string mismatch.

## Variant phrasings
### robotparser can_fetch returns False on allowed URL
Same problem: the agent string passed to can_fetch does not match any User-agent block, so the default deny applies. Align the strings.
### RobotFileParser says disallowed when robots.txt allows it
Check scheme (http vs https) on set_url and re-read before can_fetch. Stale or wrong-scheme rules are the usual cause.

## Tool/version compatibility
urllib.robotparser in Python 3.8 through 3.12 behaves the same here. Does not follow redirects when reading robots.txt. For production crawlers, consider protector or reppy, which handle wildcards and redirects more predictably.

## Why it happens
robotparser matches rules to the agent string given to can_fetch, not the one in your HTTP headers. If that string matches no User-agent block, the parser falls back to the default entry, which defaults to disallowing everything. Separately, the parser fetches robots.txt from the exact URL given to set_url, so an http fetch on a site that serves its real robots.txt on https returns rules (or nothing) for the wrong origin.

## Edge cases / pitfalls
- A trailing wildcard (Disallow: /private*) is not standard robots syntax; urllib matches it literally in some versions.
- robots.txt is per-host including port: example.com:8080 has different rules than example.com.
- Respect the file: treat consistent False results as a signal to check your agent string and scheme first, and to use an official API if the site offers one.
- Crawl-delay is parsed but not enforced by urllib; honor it yourself.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_zlaODulYa6ItMjqqb7er1w