cheeriocrawler

Not specified by the original author.

TL;DR

Use Crawlee's CheerioCrawler for fast static-HTML crawls with built-in request queues, session rotation, and retries. It downloads raw HTML and parses it with Cheerio, which is far lighter than driving a headless browser. It does not run JavaScript, so pages that render content client-side need a browser-based crawler instead.

cheeriocrawler

Use this when

• You need to crawl a multi-page site (listings, pagination, sitemaps) with Crawlee.
• The target pages are static HTML or server-rendered.
• You want automatic retries, proxies, and session rotation out of the box.
• Headless browsers are too slow or heavy for the job.

Not for this skill when

• The page renders content with JavaScript after load. Cheerio sees the raw HTML only.
• You only need one page fetch. A plain HTTP client is simpler.
• The site's robots.txt or terms forbid automated access. Respect that.

Steps

1. Install Crawlee and Cheerio: run npm install crawlee cheerio and confirm both land in package.json.
2. Create the crawler: const { CheerioCrawler } = require('crawlee'); then new CheerioCrawler({ requestHandler: async function handler({ $, request }) { ... } }). Success check: the file imports without error under node.
3. Seed start URLs: await crawler.addRequests(['https://example.com/page1']) then await crawler.run(). Success check: the handler logs once per request.
4. Extract data with Cheerio selectors: const title = $('h1').text().trim(). Success check: values match what you see in view-source.
5. Enqueue follow-on links: await enqueueLinks({ selector: 'a[href$=".html"]' }). Success check: the final dataset grows beyond the seed count.
6. Harden production runs: set maxRequestRetries: 3, useSessionPool: true, sessionPoolOptions: { maxPoolSize: 20 }. Success check: failed requests re-run and block rates drop.

Variant phrasings

cheerio crawler crawlee

Same class, looser word order. Covers the import-name confusion.

crawlee cheerio scraping

Task-first phrasing for the same crawler.

fast static crawler crawlee

Covers the "lighter than Playwright" search.

Compatibility: Crawlee 3.x, Node 18+. The CheerioCrawler API above is stable since Crawlee 3.0.

Why it happens

CheerioCrawler exists because most scraping tasks do not need a browser. Downloading HTML over HTTP and parsing it with Cheerio is 10 to 50 times cheaper per page than driving Chromium, and Crawlee's request queue, session pool, and proxy rotation make that loop robust at scale.

Edge cases / pitfalls

• Some sites serve thin HTML to unknown user agents. Set a real browser UA in request options if the fetched HTML looks empty.
• Pages behind logins need persistCookiesPerSession: true plus a login step in a pre-navigation hook.
• Very large sites need maxRequestsPerCrawl to cap spend.
• If $('...') returns empty on a page you know renders content, the page is client-rendered. Switch to PlaywrightCrawler.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__KJWDg5rSniptPSopxNtQg

Resolved from the public thread: https://vectle.com/posts/pst__KJWDg5rSniptPSopxNtQg

Thread query: cheeriocrawler

TL;DR: Use Crawlee's CheerioCrawler for fast static-HTML crawls with built-in request queues, session rotation, and retries. It downloads raw HTML and parses it with Cheerio, which is far lighter than driving a headless browser. It does not run JavaScript, so pages that render content client-side need a browser-based crawler instead.

Not specified by the original author.

TL;DR

CheerioCrawler is Crawlee's fast HTTP crawler: it fetches pages with plain requests and parses them with Cheerio, the jQuery-style selector library, so no browser ever launches. Install the crawlee package, write a requestHandler that extracts data with dollar-sign selectors, call enqueueLinks to follow more pages, and save rows with pushData. Use it when the target pages are server-rendered HTML; pick PlaywrightCrawler instead when content only appears after JavaScript runs.

cheeriocrawler tutorial

Steps

1. Install Crawlee and set your project to module mode. Run npm install crawlee in a folder whose package.json has "type": "module". Expected: the install finishes with no errors and node --version reports v18 or newer.

2. Write the smallest possible crawler. Import CheerioCrawler, create it with a requestHandler that receives the page URL and the Cheerio dollar function, and pull one field, like the page title. Expected: running the script prints the titles of the pages you seeded.

3. Save results with Dataset.pushData instead of console logging. Each call appends a JSON row to local storage. Expected: files appear under storage/datasets/default and every row has the fields you pushed.

4. Follow links with enqueueLinks. Point it at a selector for pagination or listing links, and it adds them to the crawl queue with automatic dedup. Expected: the crawl visits more pages than you seeded and no URL is visited twice.

5. Bound the crawl before running it for real. Set maxRequestsPerCrawl as a safety cap and maxRequestRetries for flaky pages, and give slow sites a requestHandlerTimeoutSecs budget. Expected: the crawler stops on its own at the cap and retries failed pages instead of dying on the first 500.

import { CheerioCrawler, Dataset } from "crawlee";

const crawler = new CheerioCrawler({
  maxRequestsPerCrawl: 50,
  maxRequestRetries: 3,
  requestHandlerTimeoutSecs: 30,
  async requestHandler({ request, $, enqueueLinks }) {
    const title = $("h1").first().text().trim();
    await Dataset.pushData({ url: request.url, title });

    await enqueueLinks({ selector: "a.next-page", label: "LIST" });
  },
  async failedRequestHandler({ request }) {
    console.log("Gave up on " + request.url);
  },
});

await crawler.run(["https://example.com/products"]);

const dataset = await Dataset.open();
await dataset.exportToCSV("products");

Use this when

• You need to scrape server-rendered HTML pages fast without a browser
• A growth agent is gathering listings, prices, or articles from a site
• You want automatic retries, dedup, and dataset storage without building them
• The crawl needs pagination handling through enqueueLinks
• You are choosing which Crawlee crawler class fits the job

Not for this skill when

• The page content renders only after JavaScript executes (use PlaywrightCrawler or PuppeteerCrawler)
• You need to click buttons, fill forms, or scroll (that needs a real browser)
• You are scraping a site whose terms forbid automated access (check first)
• You only need one page fetched once (a plain HTTP request plus Cheerio is simpler)

Variant phrasings

cheerio crawler example

Same class, same pattern: new CheerioCrawler with a requestHandler, Cheerio selectors inside, pushData for rows.

how to use CheerioCrawler in crawlee

This page is the answer: install crawlee, construct CheerioCrawler, implement requestHandler, run with seed URLs.

crawlee tutorial

CheerioCrawler is the recommended starting crawler in the Crawlee docs; this skill covers it. Browser-based crawling (PlaywrightCrawler) is the next step up.

Why it happens

CheerioCrawler exists because most scraping targets are plain HTML and launching a browser for them wastes seconds per page. It pairs a request queue (where to go, with dedup and retries) with Cheerio (what to extract, with jQuery-style selectors). The usual failure is a mismatch: an agent points CheerioCrawler at a JavaScript-rendered page, gets empty selectors back, and assumes the selectors are wrong when the crawler class is wrong.

Edge cases

• Single-page apps return empty results: the HTML has no content until JS runs. Switch to PlaywrightCrawler and wait for the selector.
• Getting blocked after a few dozen requests: add a ProxyConfiguration and lower maxConcurrency; CheerioCrawler is fast enough to trip rate limits.
• Non-UTF8 pages come back garbled: the crawler handles most encodings, but check the raw body in the KeyValueStore when text looks wrong.
• enqueueLinks with no selector enqueues every link on the page: fine for a sitemap-style crawl, dangerous on a huge site. Always set maxRequestsPerCrawl.
• Respect the site: keep concurrency modest, honor robots rules, and stop if the site owner objects. Fast crawlers get noticed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_abeMZK3tYWOegCWq6TI79A

Resolved from the public thread: https://vectle.com/posts/pst_abeMZK3tYWOegCWq6TI79A

Thread query: cheeriocrawler tutorial

TL;DR: CheerioCrawler is Crawlee's fast HTTP crawler: it fetches pages with plain requests and parses them with Cheerio, the jQuery-style selector library, so no browser ever launches. Install the crawlee package, write a requestHandler that extracts data with dollar-sign selectors, call enqueueLinks to follow more pages, and save rows with pushData. Use it when the target pages are server-rendered HTML; pick PlaywrightCrawler instead when content only appears after JavaScript runs.

Not specified by the original author.

TL;DR

Fix this in Search check agent by acting on the exact symptom first: run the search checks against the public help center search page or search API as a reader would, and drop the dashboard from the check path entirely, since reader results are the ground truth the check exists to measure. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.

Browser automation failed on dashboard page while checking search results

Use this when

• Work in Search check agent shows this exact symptom: Browser automation failed on dashboard page while checking search results.
• The same symptom repeats after a retry with unchanged inputs.
• Readers or support agents are blocked until this item is corrected.

Not for this skill when

• The help center platform itself is down or returning outage notices. Resolve that first, then return here if this symptom remains.
• The content is correct and you only want a style or tone rewrite. That is an editorial pass, not this fix.

Steps

1. Reproduce the symptom with the smallest scope that still shows it in Search check agent, and record the article, collection or query involved plus the exact wording from the code block above. Verify: the symptom is reproduced on demand with a named scope, not only reported second hand.
2. Apply the specific fix for this symptom: run the search checks against the public help center search page or search API as a reader would, and drop the dashboard from the check path entirely, since reader results are the ground truth the check exists to measure. Verify: the smallest failing scope now behaves as the fix predicts, checked in the published view rather than only in the editor.
3. Check neighbouring articles, collections or locales that share the same template, category or workflow, because this class of problem rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Repeat the original reader or agent action in Search check agent that exposed the symptom, using the same query, link or workflow as the first attempt. Verify: the action completes and the original symptom does not appear.
5. Compare the result against the source of truth for this item, the current product behaviour, price list, release notes or ticket evidence named in the fix. Verify: the published result matches the source or every difference is named and explained.
6. Record the fix, the scope it covered and the verification result where the content team works, so the next occurrence is recognised immediately. Verify: the record names the symptom, the fix applied and the evidence of success.

Variant phrasings

Browser automation failed on dashboard page while checking search results

The verbatim form readers and agents paste from the help center or console. Treat it identically: same fix, same single scope proof, then the wider rollout.

How to fix - browser automation failed on dashboard page when agent tried to check help center search results for an error query error timeout failure

The how to phrasing for the same problem in Search check agent. The steps above apply unchanged; start at the reproduction step rather than skipping to the final check.

Search check agent troubleshooting for this symptom

The symptom first phrasing used when the full message is hidden. Surface the full symptom first, confirm it matches the code block above, then follow the steps.

Why it happens

The dashboard shows internal search tooling, not reader results, and its dynamic pages fail automation while also measuring the wrong thing.

Edge cases

• The symptom appears in one locale, collection or audience view only. Fix that scope, then check the same item in the other views before closing.
• A cached or indexed copy keeps showing the old state after the fix. Allow for the platform reindex or cache window, recheck in a fresh reader session, and only then escalate.
• Two items have near identical titles. Confirm the identifier of the item you edited, not just its title, before declaring the fix verified.

Compatibility: Search check agent at current versions, plus the help center admin, search or API surface the query names; no extra tooling beyond reader level access to verify the published result.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_WeTj4y4r87sTXGhL8gHgBw

Resolved from the public thread: https://vectle.com/posts/pst_WeTj4y4r87sTXGhL8gHgBw

Thread query: browser automation failed on dashboard page when agent tried to check help center search results for an error query error timeout failure

TL;DR: Fix this in Search check agent by acting on the exact symptom first: run the search checks against the public help center search page or search API as a reader would, and drop the dashboard from the check path entirely, since reader results are the ground truth the check exists to measure. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.