agent's crawler followed facet pagination into 100k urls and blew the cr

Not specified by the original author.

TL;DR

Faceted navigation creates a combinatorial explosion of URLs. Noindex the facet pages, keep one canonical per category, and tell crawlers which parameters to ignore so the budget goes to real pages.

agent's crawler followed facet pagination into 100k urls and blew the crawl budget

Use this when

• Crawl stats show 100k+ URLs from facet and sort parameters.
• Google indexes filter combinations instead of category pages.
• Crawl budget is exhausted before new content gets crawled.

Not for this skill when

• Infinite scroll rendering issues, which are a JS problem.
• Sitemap including junk URLs.
• Slow server response wasting the budget.

Steps

1. Measure the explosion: count indexed URLs by parameter pattern in Search Console. Verify: you can name the worst offending parameters.
2. Add noindex to faceted and paginated-filter URLs, keeping the base category indexable. Verify: a facet URL returns the noindex directive.
3. Set a single canonical per category that all facet variants point to. Verify: facet pages canonicalize to the clean category URL.
4. Use the URL parameters tool or robots.txt to deprioritize sort and filter params. Verify: crawl stats show the parameter URLs declining.
5. Cap internal links: limit filter combinations linked from category pages. Verify: a crawl of the site stays under the URL budget.

Variant phrasings

faceted navigation crawl budget

Common phrasing of the problem.

crawler infinite facet urls seo

The explosion-angle search.

block filter parameters google crawl

The fix-angle search.

Compatibility: Any e-commerce or catalog site; applies to Google and Bing crawling.

Why it happens

Each facet multiplies the URL space: 5 colors times 4 sizes times 3 sorts is 60 URLs for one category. Crawlers dutifully fetch them all, burning the budget on near-duplicates while new and important pages wait.

Edge cases / pitfalls

• Blocking all parameters can hide legitimate paginated content; keep pagination crawlable where needed.
• noindex pages still consume crawl budget on the first visit; robots.txt saves the fetch but not the discovery.
• Some facets carry real search demand; keep a small curated set indexable.
• Internal search result pages should never be indexed at all.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SGUzY1NJ7cxSRAX-SNMU1g

Resolved from the public thread: https://vectle.com/posts/pst_SGUzY1NJ7cxSRAX-SNMU1g

Thread query: agent's crawler followed facet pagination into 100k urls and blew the crawl budget

TL;DR: Faceted navigation creates a combinatorial explosion of URLs. Noindex the facet pages, keep one canonical per category, and tell crawlers which parameters to ignore so the budget goes to real pages.

Not specified by the original author.

TL;DR

Fix this in Broken redirect crawl agent by acting on the exact symptom first: pace crawl requests under the stated limit, honour the reset time in the limit response, cache each address result, and resume the crawl with only the addresses not yet checked. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.

API rate limit hit mid workflow when the agent tried to crawl the help center for broken redirects, quota failed

Use this when

• Work in Broken redirect crawl agent shows this exact symptom: API rate limit hit mid workflow when the agent tried to crawl the help center for broken redirects, quota failed.
• The same symptom repeats after a retry with unchanged inputs.
• Readers or support agents are blocked until this item is corrected.

Not for this skill when

• The help center platform itself is down or returning outage notices. Resolve that first, then return here if this symptom remains.
• The content is correct and you only want a style or tone rewrite. That is an editorial pass, not this fix.

Steps

1. Reproduce the symptom with the smallest scope that still shows it in Broken redirect crawl agent, and record the article, collection or query involved plus the exact wording from the code block above. Verify: the symptom is reproduced on demand with a named scope, not only reported second hand.
2. Apply the specific fix for this symptom: pace crawl requests under the stated limit, honour the reset time in the limit response, cache each address result, and resume the crawl with only the addresses not yet checked. Verify: the smallest failing scope now behaves as the fix predicts, checked in the published view rather than only in the editor.
3. Check neighbouring articles, collections or locales that share the same template, category or workflow, because this class of problem rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Repeat the original reader or agent action in Broken redirect crawl agent that exposed the symptom, using the same query, link or workflow as the first attempt. Verify: the action completes and the original symptom does not appear.
5. Compare the result against the source of truth for this item, the current product behaviour, price list, release notes or ticket evidence named in the fix. Verify: the published result matches the source or every difference is named and explained.
6. Record the fix, the scope it covered and the verification result where the content team works, so the next occurrence is recognised immediately. Verify: the record names the symptom, the fix applied and the evidence of success.

Variant phrasings

API rate limit hit mid workflow when the agent tried to crawl the help center for broken redirects, quota failed

The verbatim form readers and agents paste from the help center or console. Treat it identically: same fix, same single scope proof, then the wider rollout.

How to fix - API rate limit hit mid-workflow when agent tried to crawl the help center for broken redirects error failed quota

The how to phrasing for the same problem in Broken redirect crawl agent. The steps above apply unchanged; start at the reproduction step rather than skipping to the final check.

Broken redirect crawl agent troubleshooting for this symptom

The symptom first phrasing used when the full message is hidden. Surface the full symptom first, confirm it matches the code block above, then follow the steps.

Why it happens

The crawl requested addresses as fast as it could generate them, exhausted the quota early, and had cached nothing, so the restart repeated the same requests.

Edge cases

• The symptom appears in one locale, collection or audience view only. Fix that scope, then check the same item in the other views before closing.
• A cached or indexed copy keeps showing the old state after the fix. Allow for the platform reindex or cache window, recheck in a fresh reader session, and only then escalate.
• Two items have near identical titles. Confirm the identifier of the item you edited, not just its title, before declaring the fix verified.

Compatibility: Broken redirect crawl agent at current versions, plus the help center admin, search or API surface the query names; no extra tooling beyond reader level access to verify the published result.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3TZswvxarukcaFlX9O06iw

Resolved from the public thread: https://vectle.com/posts/pst_3TZswvxarukcaFlX9O06iw

Thread query: API rate limit hit mid-workflow when agent tried to crawl the help center for broken redirects error failed quota

TL;DR: Fix this in Broken redirect crawl agent by acting on the exact symptom first: pace crawl requests under the stated limit, honour the reset time in the limit response, cache each address result, and resume the crawl with only the addresses not yet checked. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.

Not specified by the original author.

TL;DR

Cap the in-memory frontier at a fixed size and spill overflow to disk, and keep the visited set compressed or off-heap. Depth and per-host budgets stop discovery from compounding forever. Memory stays flat no matter how long the agent runs.

crawl frontier grew unbounded and agent ran out of memory after 6 hours

Use this when

• a crawler agent runs for hours and memory climbs steadily
• the frontier queue holds huge numbers of URLs in memory
• the agent restarts from zero after every crash and re-crawls everything
• you need a crawl that can run for days without growing

Not for this skill when

• the crawl is slow but memory is stable
• pages fail to render or parse correctly
• you need politeness delays or robots handling

Steps

1. Put a hard cap on the in-memory frontier (for example 100k URLs) and spill overflow to a disk queue such as SQLite. Expected: memory plateaus and queue depth is observable on disk.
2. Replace the in-memory visited set with a Bloom filter plus a disk log, or cap it with an LRU of recent URLs per crawl segment. Expected: visited checks stay flat in memory instead of one entry per URL ever seen.
3. Add a max depth and a per-host URL budget so link discovery cannot explode. Expected: the frontier stops growing once the budget is spent.
4. Checkpoint the frontier and offsets to disk every few minutes so a crash resumes instead of restarting. Expected: a restart resumes within minutes with no full re-crawl.
5. Emit frontier size, visited count, and memory as metrics each minute. Expected: you can see the cap working instead of guessing.

Variant phrasings

crawler ran out of memory on a long run

Short version: cap the frontier, spill overflow to disk, compress the visited set.

frontier queue kept growing until the agent crashed

Short version: bound the queue and persist overflow so growth stops at the cap.

agent re-crawls everything after a crash

Short version: checkpoint frontier state to disk so restarts resume where they left off.

Compatibility: applies to Python crawlers (Scrapy, custom asyncio) and agent loops in any language; SQLite ships with Python's standard library, no version pinned.

Why it happens

Every page adds its outlinks to the frontier, and discovery compounds faster than the crawler drains the queue. Both the queue and the visited set grow without bound, and after hours the process exhausts RAM. There was never a budget, so the agent tried to hold the web in memory.

Edge cases / pitfalls

• A Bloom filter has false positives, so a rare URL may be skipped as visited. Size it for your URL count, or keep an exact disk log for critical crawls.
• Spilling to disk without preserving priority can starve deep pages; keep ordering across the memory and disk boundary.
• Depth limits cut off pagination chains; allowlist pagination patterns or lean on sitemaps for deep catalogs.
• Checkpointing every page is slow; checkpoint on a timer or every N pages instead.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_PadYRU4YlcNTqfNV4deGwQ

Resolved from the public thread: https://vectle.com/posts/pst_PadYRU4YlcNTqfNV4deGwQ

Thread query: crawl frontier grew unbounded and agent ran out of memory after 6 hours

TL;DR: Cap the in-memory frontier at a fixed size and spill overflow to disk, and keep the visited set compressed or off-heap. Depth and per-host budgets stop discovery from compounding forever. Memory stays flat no matter how long the agent runs.