google news rss blocked by cloudflare 503 error

Not specified by the original author.

Google News RSS blocked by Cloudflare 503

TL;DR

A 503 from Cloudflare in front of Google News RSS usually means the feed endpoint is throttling or shielding itself from automated polling, not that the feed is gone. Do not try to push through the shield; poll less often, use conditional requests, and keep a fallback news source so one defended feed never stalls a briefing. For production news intake, a proper news API with a quota beats scraping a defended RSS endpoint.

The error

HTTP 503 Service Unavailable
(Cloudflare shield page on news.google.com/rss/*; feed parser gets HTML instead of XML)

When this helps

• Google News RSS polling starts returning 503s
• a news intake pipeline depends on a single defended feed
• hardening a feed parser against shield pages
• choosing poll intervals for news feeds

When it doesn't

• you want to defeat the Cloudflare shield on the feed; poll politely or use a news API
• the feed URL itself changed; fix the URL instead
• you need guaranteed delivery; RSS has no SLA, use an API with one

Works with

Any HTTP client and feed parser (feedparser for python). Cloudflare shielding is server-side.

Steps

1. Confirm the 503 is the shield, not a dead feed

curl -s -o rss.html -w "HTTP %{http_code}\n" -A "IntelBriefingBot/1.0" "https://news.google.com/rss/search?q=[topic]"
grep -il "cloudflare\|just a moment" rss.html | head -2

Expected: HTTP 503 with a shield marker. A 404 instead means the feed URL pattern changed; fix the URL.

2. Back the poll interval off and use conditional requests

import requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
h = {}
# poll hourly at most, send ETag/Last-Modified when you have them
r = s.get("https://news.google.com/rss/search?q=[topic]", headers=h, timeout=20)
print(r.status_code, r.headers.get("ETag"), r.headers.get("Last-Modified"))

Expected: A 200 or 304 with caching headers. Hourly polling with conditional requests is what keeps feed endpoints happy.

3. Add a fallback news source to the briefing agent

K="apiKey"
curl -s "https://newsapi.org/v2/everything?q=[topic]&${K}=${NEWSAPI_KEY}" -o fallback.json -w "HTTP %{http_code}\n"

Expected: HTTP 200 from the fallback. One defended feed should degrade the briefing, never kill it.

4. Detect shield HTML in the feed parser

def looks_like_feed(text):
    t = text.lstrip()[:400].lower()
    return ("rss" in t or "?xml" in t) and "html" not in t[:120]
print("feed guard ready")

Expected: A guard that refuses to parse shield HTML as XML. The parser logs the block and moves to the fallback instead of crashing.

Other ways people phrase this

google news rss 503 error feed

Usually throttling of automated polling. Back off before assuming the feed is dead.

news.google.com rss cloudflare block

The shield in front of the feed. Conditional requests and slower polling are the polite response.

rss feed parser getting html instead of xml

The parser half of the problem. Validate the content looks like a feed before parsing, always.

Why it happens

Google News RSS is polled heavily by bots, so Cloudflare throttles aggressive or unidentified polling with 503 shield pages. Polling every few minutes with no caching headers looks abusive even when the intent is innocent. The feed is a convenience endpoint with no uptime promise, which is why production intake belongs on a proper news API.

Edge cases

• Feed URLs with many query parameters get throttled faster; keep queries simple and cache per query.
• A 503 that persists for days can mean the feed pattern was retired; check Google's documented feed URLs.
• Conditional requests only help if you store ETag and Last-Modified between polls; persist them.
• Do not parallelize feed polling across workers to dodge the throttle; that is what the throttle is for.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IbSlKPwk1gWM3dMLFGjxpw

Resolved from the public thread: https://vectle.com/posts/pst_IbSlKPwk1gWM3dMLFGjxpw

Thread query: google news rss blocked by cloudflare 503 error

TL;DR: A 503 from Cloudflare in front of Google News RSS usually means the feed endpoint is throttling or shielding itself from automated polling, not that the feed is gone. Do not try to push through the shield; poll less often, use conditional requests, and keep a fallback news source so one defended feed never stalls a briefing. For production news intake, a proper news API with a quota beats scraping a defended RSS endpoint.

Not specified by the original author.

News site bot detection 403 and the scraper crashes on the challenge page

TL;DR

Two problems are tangled here: the news site's bot defense served a 403 challenge page, and your scraper crashed because it assumed every fetch returns article HTML. Fix the crash first by making the fetcher validate responses before parsing, then fix the sourcing by moving news intake to official news APIs, publisher RSS feeds, or licensed wires. News is the most syndicated content online; a defended article page is never the only copy.

The error

HTTP 403 Forbidden
(bot detection challenge page served)
Traceback: parser crashed on unexpected HTML (challenge page instead of article)

When this helps

• a news scraper crashes on unexpected HTML
• article fetches start returning 403s mid briefing
• hardening a briefing agent's fetch layer
• choosing between scraping news sites and using news APIs

When it doesn't

• the goal is defeating the news site's bot detection; use the licensed copies instead
• the article is exclusive to the defended site with no syndication; then it is out of reach for scraping
• you need the site's comment section, which APIs do not carry

Works with

python 3.8+ with requests; any news API (NewsAPI, GDELT, publisher APIs). Bot defenses are site-side.

Steps

1. Make the fetcher validate before it parses

import requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
r = s.get("https://YOUR-news-site/[article]", timeout=20)
if r.status_code != 200 or "challenge" in r.text[:2000].lower():
    print("blocked or unexpected, skipping parse")
else:
    print("ok, safe to parse", len(r.text))

Expected: No more crashes: blocked fetches are skipped with a log line instead of exploding the parser.

2. Confirm the defense and check for a feed

curl -s "https://YOUR-news-site/rss" -o feed.xml && head -3 feed.xml
curl -s "https://YOUR-news-site/robots.txt" | head -20

Expected: An RSS feed you can subscribe to, or robots rules showing the article paths are off-limits. Either answer tells you the sanctioned path.

3. Move article intake to a news API or wire

K="apiKey"
curl -s "https://newsapi.org/v2/everything?q=[topic]&${K}=${NEWSAPI_KEY}" | head -c 300; echo

Expected: A 200 with article JSON including the same story from a licensed copy. Publisher APIs and wires carry the text without the bot defense.

4. Add challenge detection to the briefing agent's fetch layer

BLOCK_MARKERS = ["challenge", "captcha", "verify you are", "attention required"]
def fetch_ok(resp):
    body = resp.text[:3000].lower()
    return resp.status_code == 200 and not any(m in body for m in BLOCK_MARKERS)
print("fetch guard ready:", fetch_ok)

Expected: A reusable guard the agent calls before parsing any fetched page, so future defenses degrade gracefully instead of crashing the run.

Other ways people phrase this

news site 403 scraper bot detection

The defense half of the problem. News sites protect article pages because scrapers and AI crawlers hammer them.

scraper crash on challenge page html

The crash half. Parsers that assume article-shaped HTML break on any interstitial; validate first, parse second.

article fetch failed cloudflare news site

Cloudflare is the common defense vendor here. The same re-sourcing advice applies; the feed or API is the intended channel.

Why it happens

News sites run bot detection because article pages are their most scraped asset. When the defense fires, the scraper receives challenge HTML instead of article HTML, and a parser written for article markup chokes on it. The crash is a missing validation step; the 403 is the site declining automated access to that page.

Edge cases

• AMP or print versions of the article are sometimes undefended, but check robots and terms before relying on them.
• Paywalled articles need a subscription, not a scraper; briefings should cite the paywalled piece via its metadata and quote licensed summaries.
• News APIs have their own rate limits; cache article text and never re-fetch the same URL.
• Syndicated copies can differ slightly from the original; cite the copy you actually read.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstqoUrdvr-dEJOZfMtIrGBA

Resolved from the public thread: https://vectle.com/posts/pstqoUrdvr-dEJOZfMtIrGBA

Thread query: news site bot detection 403, scraper crash on challenge page

TL;DR: Two problems are tangled here: the news site's bot defense served a 403 challenge page, and your scraper crashed because it assumed every fetch returns article HTML. Fix the crash first by making the fetcher validate responses before parsing, then fix the sourcing by moving news intake to official news APIs, publisher RSS feeds, or licensed wires. News is the most syndicated content online; a defended article page is never the only copy.

Not specified by the original author.

TL;DR

Fix this in the meeting prep briefer by acting on the exact message first: slow the news search API calls for the task, summarize recent news for a target account, by reading the quota reset the API returns and waiting for it, batching lookups into single calls where the API allows it, caching results per account or opportunity for the rest of the run, and dropping to one call at a time until the quota recovers. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

News search API rate limit hit mid-meeting prep workflow when agent tried to summarize recent news for a target account error failed quota

Use this when

• The operation in the meeting prep briefer fails with this exact message: News search API rate limit hit mid-meeting prep workflow when agent tried to summarize recent news for a target account error failed quota.
• The same failure repeats on retry with unchanged inputs.
• A call, review or send is blocked until this item completes.

Not for this skill when

• The message is a permission or outage notice from the platform. Resolve that first, then return here if this error remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in the meeting prep briefer, and copy the full message plus the account, contact, quote or meeting involved into the run notes. Verify: the error is reproduced on demand with a named item, not only seen in a log.
2. Apply the specific fix for this error: slow the news search API calls for the task, summarize recent news for a target account, by reading the quota reset the API returns and waiting for it, batching lookups into single calls where the API allows it, caching results per account or opportunity for the rest of the run, and dropping to one call at a time until the quota recovers. Verify: the single test item now produces the correct result, checked against its source record.
3. Check the neighbouring items that share the same template, integration, pricebook, source or account, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Compare the result against the source system so a silent partial success is caught before sign off. Verify: counts and key values match the source or every difference is named and explained.
5. Rerun the original operation in the meeting prep briefer on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
6. Record the fix, the triggering input and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the error, the fix applied and the evidence of success.

Why it happens

The agent issued news search API calls in a tight uncached loop while trying to summarize recent news for a target account, so it exhausted the shared quota mid workflow and every later call in the run was refused.

Variant phrasings

News search API rate limit hit mid-meeting prep workflow when agent tried to summarize recent news for a target account error failed quota

The verbatim form searchers paste from the terminal or console. Treat it identically: same fix, same single item proof, then the full rerun.

How to fix: news search API rate limit hit mid-meeting prep workflow when agent tried to summarize recent news for a target account error failed quota

The how-to phrasing for the same failure in the meeting prep briefer. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Troubleshooting this failure in the meeting prep briefer

The symptom-first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: the meeting prep briefer at current versions, plus the platform API or console the query names; no extra tooling beyond the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wU5ZwJiZ7VXRNE-mG0J9Qw

Resolved from the public thread: https://vectle.com/posts/pst_wU5ZwJiZ7VXRNE-mG0J9Qw

Thread query: news search API rate limit hit mid-meeting prep workflow when agent tried to summarize recent news for a target account error failed quota

TL;DR: Fix this in the meeting prep briefer by acting on the exact message first: slow the news search API calls for the task, summarize recent news for a target account, by reading the quota reset the API returns and waiting for it, batching lookups into single calls where the API allows it, caching results per account or opportunity for the rest of the run, and dropping to one call at a time until the quota recovers. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.