robots.txt cache went stale and agent's crawler fetched disallowed paths
Shows how to refresh robots.txt on a TTL and check every fetch against it. Use it when a crawler hits disallowed paths because its cached rules expired. Not for parsing robots.txt syntax from scratch or for legal advice.
TL;DR
Cache robots.txt per host with a short TTL (one hour is a safe default) and re-fetch it when the TTL expires or at the start of every crawl run, whichever comes first. Check each URL against the fresh rules before fetching. Site owners change their rules; a week-old cache is a guess, and fetching disallowed paths burns trust you need.
robots.txt cache went stale and agent's crawler fetched disallowed pathsUse this when
- the crawler fetched paths the current robots.txt disallows
- the robots cache has no expiry
- rules changed upstream but the crawler kept the old ones
Not for this skill when
- you need to learn robots.txt syntax basics
- the site has no robots.txt at all
- you want legal guidance on scraping (talk to counsel)
Steps
- Store robots.txt per host with a fetched-at timestamp. Verify: the cache entry shows when it was fetched.
- Set TTL to 3600 seconds; on expiry, re-fetch before the next request to that host. Verify: an expired entry triggers exactly one re-fetch.
- Also force a refresh at the start of every crawl run regardless of TTL. Verify: a run that starts with a 2-day-old cache re-fetches first.
- Check every candidate URL against the rules before fetching; drop disallowed URLs from the queue. Verify: a disallowed URL never reaches the HTTP client.
- Log disallowed skips separately so you can see what the rules excluded. Verify: the skip log is non-empty on a typical crawl.
Tool/version compatibility
Python's urllib robotparser, the robotexclusionrulesparser package, or any robots parser. TTL logic is a few lines anywhere.
Why it happens
Robots.txt was fetched once and cached forever. The crawler kept obeying rules that no longer matched the live file, in both directions.
Edge cases / pitfalls
- A 404 on the robots fetch means no rules (proceed); a 5xx means keep the old rules and back off.
- Crawl-delay directives deserve the same respect as disallows; feed them into your pacing.
- Respecting robots.txt is the floor of good citizenship, not a ceiling; aggressive-but-allowed crawling can still get you blocked.
Variant phrasings
Crawler ignored updated robots.txt rules
Short version: stale robots cache.
Agent fetched pages the site had newly disallowed
Short version: no robots refresh.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstDsHGFWEt-RYxCuLRVRt1g