How the Reader gets past Cloudflare: cache salvage and bot user agents

The Reader does not beat Cloudflare with stealth plugins alone. Per the maintainer, it falls back to a salvage path: query the Google web cache (now discontinued) or the Web Archive for the page, and set the user agent to a well-known bot like Slackbot, GPTBot, or GoogleSpider, which many sites let through. If you are building your own reader and stealth fails on a protected page, try the same ladder: fetch from the Web Archive first, then retry with a bot user agent before burning effort on captcha solving.

Context: Issue jina-ai/reader#66 (closed, 7 comments): a user asked how the Jina Reader gets past Cloudflare protection on heavily guarded pages when puppeteer-extra-plugin-stealth alone is not enough, since the source shows no captcha solving. A maintainer explained the actual mechanisms.

Source: Issue jina-ai/reader#66 (closed, 7 comments): a user asked how the Jina Reader gets past Cloudflare protection on heavily guarded pages when puppeteer-extra-plugin-stealth alone is not enough, since the source shows no captcha solving. A maintainer explained the actual mechanisms.

Verified resolution: The Reader does not beat Cloudflare with stealth plugins alone. Per the maintainer, it falls back to a salvage path: query the Google web cache (now discontinued) or the Web Archive for the page, and set the user agent to a well-known bot like Slackbot, GPTBot, or GoogleSpider, which many sites let through. If you are building your own reader and stealth fails on a protected page, try the same ladder: fetch from the Web Archive first, then retry with a bot user agent before burning effort on captcha solving.

Source URL: https://github.com/jina-ai/reader/issues/66