## TL;DR
Faceted navigation creates a combinatorial explosion of URLs. Noindex the facet pages, keep one canonical per category, and tell crawlers which parameters to ignore so the budget goes to real pages.

```text
agent's crawler followed facet pagination into 100k urls and blew the crawl budget
```

## Use this when
- Crawl stats show 100k+ URLs from facet and sort parameters.
- Google indexes filter combinations instead of category pages.
- Crawl budget is exhausted before new content gets crawled.

## Not for this skill when
- Infinite scroll rendering issues, which are a JS problem.
- Sitemap including junk URLs.
- Slow server response wasting the budget.

## Steps
1. Measure the explosion: count indexed URLs by parameter pattern in Search Console. Verify: you can name the worst offending parameters.
2. Add noindex to faceted and paginated-filter URLs, keeping the base category indexable. Verify: a facet URL returns the noindex directive.
3. Set a single canonical per category that all facet variants point to. Verify: facet pages canonicalize to the clean category URL.
4. Use the URL parameters tool or robots.txt to deprioritize sort and filter params. Verify: crawl stats show the parameter URLs declining.
5. Cap internal links: limit filter combinations linked from category pages. Verify: a crawl of the site stays under the URL budget.

## Variant phrasings
### faceted navigation crawl budget
Common phrasing of the problem.
### crawler infinite facet urls seo
The explosion-angle search.
### block filter parameters google crawl
The fix-angle search.

Compatibility: Any e-commerce or catalog site; applies to Google and Bing crawling.

## Why it happens
Each facet multiplies the URL space: 5 colors times 4 sizes times 3 sorts is 60 URLs for one category. Crawlers dutifully fetch them all, burning the budget on near-duplicates while new and important pages wait.

## Edge cases / pitfalls
- Blocking all parameters can hide legitimate paginated content; keep pagination crawlable where needed.
- noindex pages still consume crawl budget on the first visit; robots.txt saves the fetch but not the discovery.
- Some facets carry real search demand; keep a small curated set indexable.
- Internal search result pages should never be indexed at all.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SGUzY1NJ7cxSRAX-SNMU1g