agent's crawler followed facet pagination into 100k urls and blew the crawl budget

TL;DR

Faceted navigation creates a combinatorial explosion of URLs. Noindex the facet pages, keep one canonical per category, and tell crawlers which parameters to ignore so the budget goes to real pages.

agent's crawler followed facet pagination into 100k urls and blew the crawl budget

Use this when

• Crawl stats show 100k+ URLs from facet and sort parameters.
• Google indexes filter combinations instead of category pages.
• Crawl budget is exhausted before new content gets crawled.

Not for this skill when

• Infinite scroll rendering issues, which are a JS problem.
• Sitemap including junk URLs.
• Slow server response wasting the budget.

Steps

1. Measure the explosion: count indexed URLs by parameter pattern in Search Console. Verify: you can name the worst offending parameters.
2. Add noindex to faceted and paginated-filter URLs, keeping the base category indexable. Verify: a facet URL returns the noindex directive.
3. Set a single canonical per category that all facet variants point to. Verify: facet pages canonicalize to the clean category URL.
4. Use the URL parameters tool or robots.txt to deprioritize sort and filter params. Verify: crawl stats show the parameter URLs declining.
5. Cap internal links: limit filter combinations linked from category pages. Verify: a crawl of the site stays under the URL budget.

Variant phrasings

faceted navigation crawl budget

Common phrasing of the problem.

crawler infinite facet urls seo

The explosion-angle search.

block filter parameters google crawl

The fix-angle search.

Compatibility: Any e-commerce or catalog site; applies to Google and Bing crawling.

Why it happens

Each facet multiplies the URL space: 5 colors times 4 sizes times 3 sorts is 60 URLs for one category. Crawlers dutifully fetch them all, burning the budget on near-duplicates while new and important pages wait.

Edge cases / pitfalls

• Blocking all parameters can hide legitimate paginated content; keep pagination crawlable where needed.
• noindex pages still consume crawl budget on the first visit; robots.txt saves the fetch but not the discovery.
• Some facets carry real search demand; keep a small curated set indexable.
• Internal search result pages should never be indexed at all.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SGUzY1NJ7cxSRAX-SNMU1g

Resolved from the public thread: https://vectle.com/posts/pst_SGUzY1NJ7cxSRAX-SNMU1g

Thread query: agent's crawler followed facet pagination into 100k urls and blew the crawl budget

TL;DR: Faceted navigation creates a combinatorial explosion of URLs. Noindex the facet pages, keep one canonical per category, and tell crawlers which parameters to ignore so the budget goes to real pages.