Empty content on referer-checked PDF links - pass a referer to the Reader

TL;DR

Add noCache true so you do not get a stale empty result. If the Reader returns empty content for a PDF link from academia.edu or a similar referer-checking site, pass a referer explicitly.

Steps

1. If the Reader returns empty content for a PDF link from academia.edu or a similar referer-checking site, pass a referer explicitly. In POST mode put referer in the JSON body next to the url (scholar.google.com works for academia.edu), or send it as a request header in GET mode. Add noCache true so you do not get a stale empty result. If the page has a real anti-bot wall like a human-verification challenge, no parameter will help - fetch it another way.

Expected: A maintainer confirmed anti-crawler pages cannot be fixed server-side, but added referer support.

When to use

You are seeing this: In POST mode put referer in the JSON body next to the url (scholar.google.com works for academia.edu), or send it as a request header in GET mode. Use this skill when you run into "Empty content on referer-checked PDF links - pass a referer to the Reader".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: Issue jina-ai/reader#106 (closed, 7 comments): PDF links from ResearchGate and academia.edu kept returning empty content or a verify-you-are-human page through the Reader API. Some of these sites check the HTTP referer - one reporter showed a curl that only works with -e set to scholar.google.com. A maintainer confirmed anti-crawler pages cannot be fixed server-side, but added referer support.

Verified resolution: If the Reader returns empty content for a PDF link from academia.edu or a similar referer-checking site, pass a referer explicitly. In POST mode put referer in the JSON body next to the url (scholar.google.com works for academia.edu), or send it as a request header in GET mode. Add noCache true so you do not get a stale empty result. If the page has a real anti-bot wall like a human-verification challenge, no parameter will help - fetch it another way.

Source URL: https://github.com/jina-ai/reader/issues/106