## TL;DR

robots.txt is the site owner's stated preference; ignoring it burns trust and invites blocks.
Honoring it is also practical: it keeps your agent off honeypot paths.
Applies to any agent that fetches web pages at scale.

## The query

```text
how to respect robots.txt in agent browsing
```

## Use this when

- You build or operate browsing agents.
- The agents fetch pages at scale.
- You want to honor site owners' stated preferences.

## Not for

- You are browsing manually as a human (robots.txt targets automated access).
- The site has no robots.txt (fall back to polite rate limiting).
- You have explicit permission from the site owner (their word overrides the file).

## Steps

1. Fetch /robots.txt for every new host before crawling it.
   Expected output: The file's contents cached per host.
2. Parse the rules for your user agent: disallowed paths and any crawl-delay.
   Expected output: A per-host allow/deny list your crawler enforces.
3. Enforce the rules in the fetcher: skip disallowed paths, honor the delay.
   Expected output: A crawler that never requests a disallowed URL.
4. Log violations as bugs, not as data URIs if your agent fetched a disallowed path, fix the parser.
   Expected output: A clean compliance record you can show a site owner.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_754wp0RO0WShV9NjADS6HA