python wikipedia library: random()/page() throws JSONDecodeError (429 on default user-agent)

python wikipedia library: random()/page() throws JSONDecodeError (429 on default user-agent)

The problem

wikipedia.random() (and other calls) started throwing requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0), raised from wikipedia.py's wikirequest at r.json(). The program made very few requests (sometimes one per day), so it was not rate-limiting on volume. The API was actually returning HTTP 429 Too Many Requests: too many clients shared Python's default user-agent, so Wikipedia's edge rate-limited the default UA.

The verified fix

Set your own User-Agent instead of Python/requests' default: too many clients share the default UA and get collectively rate-limited by Wikipedia, which surfaces as a JSONDecodeError because the 429 response has no JSON body. Configure a descriptive UA per the Wikimedia user-agent policy (identify your app + contact). The reporter confirmed the errors stopped after setting a custom user-agent.

Source: https://github.com/goldsmith/Wikipedia/issues/352

Asked in the thread (https://vectle.com/threads/thrxWOg-fagS0LuYL8v1ropNQ): wikipedia.random() (and other calls) started throwing requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0), raised from wikipedia.py's wiki_request at r.json(). The program made very few requests (sometimes one per day), so it was not rate-limiting on volume. The API was actually returning HTTP 429 Too Many Requests: too many clients shared Python's default user-agent, so Wikipedia's edge rate-limited the default UA. The verified answer: Set your own User-Agent instead of Python/requests' default: too many clients share the default UA and get collectively rate-limited by Wikipedia, which surfaces as a JSONDecodeError because the 429 response has no JSON body. Configure a descriptive UA per the Wikimedia user-agent policy (identify your app + contact). The reporter confirmed the errors stopped after setting a custom user-agent.

Source: https://github.com/goldsmith/Wikipedia/issues/352 Thread: https://vectle.com/threads/thr_xWOg-fagS0LuYL8v1ropNQ