How the Reader gets past Cloudflare: cache salvage and bot user agents

TL;DR

The Reader does not beat Cloudflare with stealth plugins alone. Per the maintainer, it falls back to a salvage path: query the Google web cache (now discontinued) or the Web Archive for the page, and set the user agent to a well-known bot like Slackbot, GPTBot, or GoogleSpider, which many sites let through.

Fix

The Reader does not beat Cloudflare with stealth plugins alone. Per the maintainer, it falls back to a salvage path: query the Google web cache (now discontinued) or the Web Archive for the page, and set the user agent to a well-known bot like Slackbot, GPTBot, or GoogleSpider, which many sites let through. If you are building your own reader and stealth fails on a protected page, try the same ladder: fetch from the Web Archive first, then retry with a bot user agent before burning effort on captcha solving.

Context: Issue jina-ai/reader#66 (closed, 7 comments): a user asked how the Jina Reader gets past Cloudflare protection on heavily guarded pages when puppeteer-extra-plugin-stealth alone is not enough, since the source shows no captcha solving. A maintainer explained the actual mechanisms.

Matched source

Source: Published skill Original query: "How the Reader gets past Cloudflare: cache salvage and bot user agents" Key terms: agents, cache, cloudflare, gets, past, reader, salvage, user

When to use this

• Use this when your task, error, or question matches the title above and the setup in the Fix section.

When not to use this

• Skip it when the error text, tool, or situation is different. Forcing a mismatched fix usually costs more time than searching again.

Compatibility

The original write-up does not pin tool versions. Treat the commands and UI paths above as reflecting the author's setup, and check your tool's own docs if your version looks different.

Source: Issue jina-ai/reader#66 (closed, 7 comments): a user asked how the Jina Reader gets past Cloudflare protection on heavily guarded pages when puppeteer-extra-plugin-stealth alone is not enough, since the source shows no captcha solving. A maintainer explained the actual mechanisms.

Verified resolution: The Reader does not beat Cloudflare with stealth plugins alone. Per the maintainer, it falls back to a salvage path: query the Google web cache (now discontinued) or the Web Archive for the page, and set the user agent to a well-known bot like Slackbot, GPTBot, or GoogleSpider, which many sites let through. If you are building your own reader and stealth fails on a protected page, try the same ladder: fetch from the Web Archive first, then retry with a bot user agent before burning effort on captcha solving.

Source URL: https://github.com/jina-ai/reader/issues/66