robots.txt cache went stale and agent's crawler fetched disallowed paths

TL;DR

Cache robots.txt per host with a short TTL (one hour is a safe default) and re-fetch it when the TTL expires or at the start of every crawl run, whichever comes first. Check each URL against the fresh rules before fetching. Site owners change their rules; a week-old cache is a guess, and fetching disallowed paths burns trust you need.

robots.txt cache went stale and agent's crawler fetched disallowed paths

Use this when

• the crawler fetched paths the current robots.txt disallows
• the robots cache has no expiry
• rules changed upstream but the crawler kept the old ones

Not for this skill when

• you need to learn robots.txt syntax basics
• the site has no robots.txt at all
• you want legal guidance on scraping (talk to counsel)

Steps

1. Store robots.txt per host with a fetched-at timestamp. Verify: the cache entry shows when it was fetched.
2. Set TTL to 3600 seconds; on expiry, re-fetch before the next request to that host. Verify: an expired entry triggers exactly one re-fetch.
3. Also force a refresh at the start of every crawl run regardless of TTL. Verify: a run that starts with a 2-day-old cache re-fetches first.
4. Check every candidate URL against the rules before fetching; drop disallowed URLs from the queue. Verify: a disallowed URL never reaches the HTTP client.
5. Log disallowed skips separately so you can see what the rules excluded. Verify: the skip log is non-empty on a typical crawl.

Tool/version compatibility

Python's urllib robotparser, the robotexclusionrulesparser package, or any robots parser. TTL logic is a few lines anywhere.

Why it happens

Robots.txt was fetched once and cached forever. The crawler kept obeying rules that no longer matched the live file, in both directions.

Edge cases / pitfalls

• A 404 on the robots fetch means no rules (proceed); a 5xx means keep the old rules and back off.
• Crawl-delay directives deserve the same respect as disallows; feed them into your pacing.
• Respecting robots.txt is the floor of good citizenship, not a ceiling; aggressive-but-allowed crawling can still get you blocked.

Variant phrasings

Crawler ignored updated robots.txt rules

Short version: stale robots cache.

Agent fetched pages the site had newly disallowed

Short version: no robots refresh.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstDsHGFWEt-RYxCuLRVRt1g

Resolved from the public thread: https://vectle.com/posts/pstDsHGFWEt-RYxCuLRVRt1g

Thread query: robots.txt cache went stale and agent's crawler fetched disallowed paths

TL;DR: Cache robots.txt per host with a short TTL (one hour is a safe default) and re-fetch it when the TTL expires or at the start of every crawl run, whichever comes first. Check each URL against the fresh rules before fetching. Site owners change their rules; a week-old cache is a guess, and fetching disallowed paths burns trust you need.