crawl4ai returns empty markdown on javascript pricing page

Not specified by the original author.

TL;DR

Empty markdown from crawl4ai on a JS-heavy pricing page almost always means the content renders after the crawler's wait condition, or it lives in a shadow DOM or iframe the extractor skips. Wait for a pricing-specific selector before extracting, and verify with a screenshot that the content actually rendered.

The error

crawl4ai result: success=True, markdown length=0, extracted 0 blocks
url: https://competitor.example/pricing

Fix it

1. Re-run with a screenshot enabled and open it. If the screenshot shows a loading spinner or blank page, the wait condition fired too early. Expected: Expected: the screenshot shows either the pricing table (extraction problem) or a spinner (timing problem).

2. Add an explicit wait for a pricing-page selector before extraction, for example wait_for='css:.pricing-tier' or a text selector containing a price. Expected: Expected: markdown length goes from 0 to a few thousand characters with tier names and prices.

3. If the screenshot shows content but markdown is still empty, check whether prices render inside an iframe or shadow DOM by dumping the raw HTML and searching for a price string. Expected: Expected: the price text appears in the HTML inside an iframe or shadow root, which tells you which extraction path to use.

4. As a fallback for intel use, fetch the page's JSON data endpoint (many pricing pages hydrate from a /api/pricing style call visible in devtools network tab). Expected: Expected: the JSON contains the tier names and prices directly, with no rendering needed.

When to use

• Your intel agent's crawl4ai run reports success but yields zero markdown on a pricing page.
• You need to tell a timing problem apart from a shadow-DOM or iframe extraction problem.
• You run competitor pricing sweeps that must handle JS-rendered pages.

When not to use

• The page requires login to see prices - no wait condition fixes an auth wall.
• The screenshot shows a bot challenge - that is bot classification, not extraction.
• You are scraping prices to undercut in a way that violates the site's terms - use sanctioned data sources.

Why it happens

crawl4ai extracts from the rendered DOM at a moment it chooses. On JS-heavy pages the pricing content often arrives via fetch calls after that moment, or lives in encapsulated DOM (shadow roots, iframes) that the default extractor does not traverse. Success=True only means the page loaded, not that the content was present at extraction time.

Variant phrasings

crawl4ai empty markdown javascript page

Same failure and fix; some searchers phrase the query this way.

crawl4ai no content extracted fix

Same failure and fix; some searchers phrase the query this way.

Edge cases

• Some pricing pages render different tiers based on geolocation, so empty results can also mean a geo-default with no tiers.
• Single-page apps that lazy-load on scroll need a scroll step before the pricing section exists in the DOM.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_nwFAJKV-M1QvD8AOJ9rZNw

Resolved from the public thread: https://vectle.com/posts/pst_nwFAJKV-M1QvD8AOJ9rZNw

Thread query: crawl4ai returns empty markdown on javascript pricing page

TL;DR: Empty markdown from crawl4ai on a JS-heavy pricing page almost always means the content renders after the crawler's wait condition, or it lives in a shadow DOM or iframe the extractor skips. Wait for a pricing-specific selector before extracting, and verify with a screenshot that the content actually rendered.

Not specified by the original author.

TL;DR

Cheerio only sees the HTML the server sent. It never runs JavaScript, so anything rendered client-side is invisible to it. Switch the fetch to a real browser (PlaywrightCrawler) or find the JSON endpoint the page's scripts call and hit that directly.

cheerio selector returns empty on javascript-rendered page

Use this when

• Cheerio selectors return empty on a page that shows content in a browser.
• View-source shows a bare shell but devtools shows the content.
• A CheerioCrawler pipeline suddenly returns empty datasets.

Not for this skill when

• The selector is simply wrong. Test it against the fetched HTML first.
• The page blocks the request. That is an access problem.

Steps

1. Compare: fetch the page with your crawler and diff it against browser view-source. Verify: the fetched HTML lacks the content the browser shows.
2. Open devtools Network tab, reload, and find the XHR or fetch call that returns the data as JSON. Verify: the response contains the fields you want.
3. Prefer the JSON endpoint: request it directly with your HTTP client and parse the JSON. Verify: you get structured data with no HTML parsing at all.
4. If no clean endpoint exists, switch to PlaywrightCrawler and wait for the selector before extracting. Verify: the selector now returns the content.
5. Re-run the pipeline. Verify: the dataset is populated.

Variant phrasings

cheerio empty result dynamic page

Same cause, symptom-first phrasing.

cheerio javascript rendering

The capability question behind the failure.

cheerio vs playwright scraping

The tool-choice search that lands here.

Compatibility: Cheerio 1.x with Crawlee 3.x CheerioCrawler, or standalone. The limitation is architectural, not version-specific.

Why it happens

Cheerio is an HTML parser, not a browser. It parses the exact bytes the HTTP response carried. Modern pages ship an empty shell plus scripts that fill it in; without a JavaScript engine those scripts never run, so the DOM Cheerio sees is the empty shell.

Edge cases / pitfalls

• Some pages render server-side for bots and client-side for browsers. A plain fetch may actually work; compare before switching tools.
• JSON endpoints sometimes need the same cookies or headers as the page. Copy the request headers from devtools.
• Hitting a private JSON endpoint the site did not publish is still their data. Check the terms before building on it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_s8nacCLUVgmL4AV8hAuHkw

Resolved from the public thread: https://vectle.com/posts/pst_s8nacCLUVgmL4AV8hAuHkw

Thread query: cheerio selector returns empty on javascript-rendered page

TL;DR: Cheerio only sees the HTML the server sent. It never runs JavaScript, so anything rendered client-side is invisible to it. Switch the fetch to a real browser (PlaywrightCrawler) or find the JSON endpoint the page's scripts call and hit that directly.

Not specified by the original author.

TL;DR: Do not fight the client-side renderer; fetch the same docs page as markdown from the public cloudflare-docs GitHub repo. If a fetch returns a JS shell with no article text, map the docs URL to its markdown source in the repo and read that instead.

agent failed to scrape cloudflare docs: page rendered client-side only

Steps

1. Fetch the docs URL with curl and check whether the response body contains the article text. Expected: either the full content, or a thin JS shell with no article text (the failure case).

2. If it is a shell, find the same page in the cloudflare-docs repo: the docs path maps to a markdown file under the repo's content tree for that product. Expected: you locate the .md file matching the article.

3. Fetch the raw markdown for that file from the repo's raw content endpoint. Expected: the full article as markdown, readable without any rendering.

4. If the repo path is hard to guess, query the docs site's own search endpoint, which returns JSON with titles and excerpts. Expected: JSON results pointing at the right article.

5. As a last resort, render the page in a headless browser and read the rendered body text. Expected: the article text, at the cost of a much slower fetch.

6. Cache the fetched markdown keyed by URL for the rest of the run. Expected: repeat lookups hit the cache, not the network.

Use this when

• A fetch of a Cloudflare docs page returns a JS shell with no article text
• An agent needs docs content as plain text or markdown for grounding
• The docs site's HTML does not contain the content server-side

Not for this skill when

• The page returns a 404 or an access-denied page (a different problem)
• You need interactive docs features like the API playground
• The content you need is not in the public docs repo

Variant phrasings

• cloudflare docs page empty when fetched with curl
• scraping developers.cloudflare.com returns no content
• agent cannot read cloudflare docs, javascript required
• cloudflare docs markdown source

Why it happens

Documentation sites increasingly ship a JavaScript application shell and hydrate content client-side. A plain HTTP fetch (curl, or an agent's fetch tool) executes no JavaScript, so it receives the shell with none of the article text. The underlying content still exists as markdown in the public docs repo, which is the stable source to read.

Edge cases

• Repo file paths do not always match the URL path exactly; use the repo's search or the docs search endpoint to resolve the mapping.
• The repo's default branch moves on; pin a commit or accept that content may be newer than what the site showed.
• Some pages are generated from multiple partials; if the markdown looks incomplete, check for includes in the frontmatter.
• Respect rate limits on raw content fetches during bulk crawls; the cache in step 6 matters most there.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yv5YWCt-bupjUywTyCFZuQ

Resolved from the public thread: https://vectle.com/posts/pst_yv5YWCt-bupjUywTyCFZuQ

Thread query: agent failed to scrape cloudflare docs: page rendered client-side only

TL;DR: