Empty content on referer-checked PDF links - pass a referer to the Reader

If the Reader returns empty content for a PDF link from academia.edu or a similar referer-checking site, pass a referer explicitly. In POST mode put referer in the JSON body next to the url (scholar.google.com works for academia.edu), or send it as a request header in GET mode. Add noCache true so you do not get a stale empty result. If the page has a real anti-bot wall like a human-verification challenge, no parameter will help - fetch it another way.

Context: Issue jina-ai/reader#106 (closed, 7 comments): PDF links from ResearchGate and academia.edu kept returning empty content or a verify-you-are-human page through the Reader API. Some of these sites check the HTTP referer - one reporter showed a curl that only works with -e set to scholar.google.com. A maintainer confirmed anti-crawler pages cannot be fixed server-side, but added referer support.

Matched source

Source: Published skill Original query: "Empty content on referer-checked PDF links - pass a referer to the Reader" Key terms: checked, content, empty, links, pass, reader, referer

Source: Issue jina-ai/reader#106 (closed, 7 comments): PDF links from ResearchGate and academia.edu kept returning empty content or a verify-you-are-human page through the Reader API. Some of these sites check the HTTP referer - one reporter showed a curl that only works with -e set to scholar.google.com. A maintainer confirmed anti-crawler pages cannot be fixed server-side, but added referer support.

Verified resolution: If the Reader returns empty content for a PDF link from academia.edu or a similar referer-checking site, pass a referer explicitly. In POST mode put referer in the JSON body next to the url (scholar.google.com works for academia.edu), or send it as a request header in GET mode. Add noCache true so you do not get a stale empty result. If the page has a real anti-bot wall like a human-verification challenge, no parameter will help - fetch it another way.

Source URL: https://github.com/jina-ai/reader/issues/106