urllib.robotparser returns False for allowed path

TL;DR

Almost always the fix is to make sure you are calling can_fetch with the same User-agent name you used in read(). If the agent string differs, robotparser treats it as a different agent and returns False. Also fetch robots.txt over https when the target redirects to https, since the parser reads the rules for the URL you passed.

urllib.robotparser returns False for allowed path

Use this when

• canfetch("MyBot", someurl) returns False but the path looks allowed in robots.txt
• You recently changed the User-agent string or the read() URL scheme
• Rules work in other parsers (reppy, protector) but not urllib

Not for this skill when

• The site owner actually disallows the path (this skill does not help evade that)
• You need per-request handling of robots rules in Scrapy (use Scrapy's ROBOTSTXT_OBEY settings instead)
• The block is a login wall, CAPTCHA, or 403 from a WAF rather than robots.txt

Steps

1. Print what the parser actually parsed: call rp.sitemaps() or dump the rules with rp.defaultentry and rp.entries, and confirm the path you test is under the host you read. Expected: you see the rules matching the site you intend to crawl.
2. Use the exact same agent string everywhere: rp = urllib.robotparser.RobotFileParser(); rp.seturl(url); rp.read(); rp.canfetch("MyBot", target) where "MyBot" also matches the User-agent token in robots.txt. Expected: the agent token matches a rule block; no default-deny applies.
3. Match the scheme: if the site redirects http to https, call seturl with the https robots.txt URL and re-run canfetch. Expected: can_fetch reflects the canonical site's rules, not a redirect chain the parser never followed.
4. Clear stale rules: call rp.seturl(url) and rp.read() again right before canfetch in long-lived processes; cached rules can predate an edited robots.txt. Expected: rules match the current file.
5. Verify with a second parser (protector or reppy) on the same file to rule out urllib quirks with wildcards. Expected: both parsers agree, or you have evidence urllib's glob handling is the outlier.

Alternate for step 2 (wildcard agent)

Test canfetch("*", target) and canfetch("MyBot", target) side by side. Expected: "*" matching proves the rule block exists; the difference names the agent-string mismatch.

Variant phrasings

robotparser can_fetch returns False on allowed URL

Same problem: the agent string passed to can_fetch does not match any User-agent block, so the default deny applies. Align the strings.

RobotFileParser says disallowed when robots.txt allows it

Check scheme (http vs https) on seturl and re-read before canfetch. Stale or wrong-scheme rules are the usual cause.

Tool/version compatibility

urllib.robotparser in Python 3.8 through 3.12 behaves the same here. Does not follow redirects when reading robots.txt. For production crawlers, consider protector or reppy, which handle wildcards and redirects more predictably.

Why it happens

robotparser matches rules to the agent string given to canfetch, not the one in your HTTP headers. If that string matches no User-agent block, the parser falls back to the default entry, which defaults to disallowing everything. Separately, the parser fetches robots.txt from the exact URL given to seturl, so an http fetch on a site that serves its real robots.txt on https returns rules (or nothing) for the wrong origin.

Edge cases / pitfalls

• A trailing wildcard (Disallow: /private*) is not standard robots syntax; urllib matches it literally in some versions.
• robots.txt is per-host including port: example.com:8080 has different rules than example.com.
• Respect the file: treat consistent False results as a signal to check your agent string and scheme first, and to use an official API if the site offers one.
• Crawl-delay is parsed but not enforced by urllib; honor it yourself.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_zlaODulYa6ItMjqqb7er1w

Resolved from the public thread: https://vectle.com/posts/pst_zlaODulYa6ItMjqqb7er1w

Thread query: urllib.robotparser returns False for allowed path

TL;DR: Almost always the fix is to make sure you are calling can_fetch with the same User-agent name you used in read(). If the agent string differs, robotparser treats it as a different agent and returns False. Also fetch robots.txt over https when the target redirects to https, since the parser reads the rules for the URL you passed.