agent's crawler followed facet pagination into 100k urls and blew the crawl budget
Contains a crawler that exploded into faceted pagination: block facet combinations, consolidate signals, and protect the crawl budget. Use when crawl stats show hundreds of thousands of near-duplicate URLs from filters and sorts. Not for sitemap bloat, which is a separate generation problem.
TL;DR
Faceted navigation creates a combinatorial explosion of URLs. Noindex the facet pages, keep one canonical per category, and tell crawlers which parameters to ignore so the budget goes to real pages.
agent's crawler followed facet pagination into 100k urls and blew the crawl budgetUse this when
- Crawl stats show 100k+ URLs from facet and sort parameters.
- Google indexes filter combinations instead of category pages.
- Crawl budget is exhausted before new content gets crawled.
Not for this skill when
- Infinite scroll rendering issues, which are a JS problem.
- Sitemap including junk URLs.
- Slow server response wasting the budget.
Steps
- Measure the explosion: count indexed URLs by parameter pattern in Search Console. Verify: you can name the worst offending parameters.
- Add noindex to faceted and paginated-filter URLs, keeping the base category indexable. Verify: a facet URL returns the noindex directive.
- Set a single canonical per category that all facet variants point to. Verify: facet pages canonicalize to the clean category URL.
- Use the URL parameters tool or robots.txt to deprioritize sort and filter params. Verify: crawl stats show the parameter URLs declining.
- Cap internal links: limit filter combinations linked from category pages. Verify: a crawl of the site stays under the URL budget.
Variant phrasings
faceted navigation crawl budget
Common phrasing of the problem.
crawler infinite facet urls seo
The explosion-angle search.
block filter parameters google crawl
The fix-angle search.
Compatibility: Any e-commerce or catalog site; applies to Google and Bing crawling.
Why it happens
Each facet multiplies the URL space: 5 colors times 4 sizes times 3 sorts is 60 URLs for one category. Crawlers dutifully fetch them all, burning the budget on near-duplicates while new and important pages wait.
Edge cases / pitfalls
- Blocking all parameters can hide legitimate paginated content; keep pagination crawlable where needed.
- noindex pages still consume crawl budget on the first visit; robots.txt saves the fetch but not the discovery.
- Some facets carry real search demand; keep a small curated set indexable.
- Internal search result pages should never be indexed at all.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_SGUzY1NJ7cxSRAX-SNMU1g