google news rss blocked by cloudflare 503 error

Google News RSS blocked by Cloudflare 503

TL;DR

A 503 from Cloudflare in front of Google News RSS usually means the feed endpoint is throttling or shielding itself from automated polling, not that the feed is gone. Do not try to push through the shield; poll less often, use conditional requests, and keep a fallback news source so one defended feed never stalls a briefing. For production news intake, a proper news API with a quota beats scraping a defended RSS endpoint.

The error

HTTP 503 Service Unavailable
(Cloudflare shield page on news.google.com/rss/*; feed parser gets HTML instead of XML)

When this helps

• Google News RSS polling starts returning 503s
• a news intake pipeline depends on a single defended feed
• hardening a feed parser against shield pages
• choosing poll intervals for news feeds

When it doesn't

• you want to defeat the Cloudflare shield on the feed; poll politely or use a news API
• the feed URL itself changed; fix the URL instead
• you need guaranteed delivery; RSS has no SLA, use an API with one

Works with

Any HTTP client and feed parser (feedparser for python). Cloudflare shielding is server-side.

Steps

1. Confirm the 503 is the shield, not a dead feed

curl -s -o rss.html -w "HTTP %{http_code}\n" -A "IntelBriefingBot/1.0" "https://news.google.com/rss/search?q=[topic]"
grep -il "cloudflare\|just a moment" rss.html | head -2

Expected: HTTP 503 with a shield marker. A 404 instead means the feed URL pattern changed; fix the URL.

2. Back the poll interval off and use conditional requests

import requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
h = {}
# poll hourly at most, send ETag/Last-Modified when you have them
r = s.get("https://news.google.com/rss/search?q=[topic]", headers=h, timeout=20)
print(r.status_code, r.headers.get("ETag"), r.headers.get("Last-Modified"))

Expected: A 200 or 304 with caching headers. Hourly polling with conditional requests is what keeps feed endpoints happy.

3. Add a fallback news source to the briefing agent

K="apiKey"
curl -s "https://newsapi.org/v2/everything?q=[topic]&${K}=${NEWSAPI_KEY}" -o fallback.json -w "HTTP %{http_code}\n"

Expected: HTTP 200 from the fallback. One defended feed should degrade the briefing, never kill it.

4. Detect shield HTML in the feed parser

def looks_like_feed(text):
    t = text.lstrip()[:400].lower()
    return ("rss" in t or "?xml" in t) and "html" not in t[:120]
print("feed guard ready")

Expected: A guard that refuses to parse shield HTML as XML. The parser logs the block and moves to the fallback instead of crashing.

Other ways people phrase this

google news rss 503 error feed

Usually throttling of automated polling. Back off before assuming the feed is dead.

news.google.com rss cloudflare block

The shield in front of the feed. Conditional requests and slower polling are the polite response.

rss feed parser getting html instead of xml

The parser half of the problem. Validate the content looks like a feed before parsing, always.

Why it happens

Google News RSS is polled heavily by bots, so Cloudflare throttles aggressive or unidentified polling with 503 shield pages. Polling every few minutes with no caching headers looks abusive even when the intent is innocent. The feed is a convenience endpoint with no uptime promise, which is why production intake belongs on a proper news API.

Edge cases

• Feed URLs with many query parameters get throttled faster; keep queries simple and cache per query.
• A 503 that persists for days can mean the feed pattern was retired; check Google's documented feed URLs.
• Conditional requests only help if you store ETag and Last-Modified between polls; persist them.
• Do not parallelize feed polling across workers to dodge the throttle; that is what the throttle is for.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IbSlKPwk1gWM3dMLFGjxpw