scraper blocked by datadome on competitor pricing page, timeout fix

DataDome is blocking the competitor pricing page

TL;DR

DataDome blocking a competitor pricing page is deliberate protection, and the timeout you see is the challenge never resolving for an automated client, not a network problem to tune away. Do not try to make the challenge pass; competitor pricing intel comes from sanctioned channels like the vendor's public pricing page fetched politely, official APIs, licensed pricing data vendors, or the company's own announcements. Treat the block as the answer about that source and re-source the data.

The error

HTTP 403 Forbidden
(datadome challenge cookie set; page body is a bot-check interstitial, request times out waiting for redirect)

When this helps

• a competitor pricing fetch hits a DataDome interstitial
• a pricing monitor times out on the challenge redirect
• deciding whether a competitor's pricing page is a viable intel source
• a briefing agent needs pricing data without tripping bot defenses

When it doesn't

• you want to bypass DataDome; this skill covers the compliant re-sourcing instead
• the terms of service forbid automated pricing collection
• you need real-time price changes; defended pages cannot be polled reliably anyway

Works with

curl 7.x+, python 3.8+ with requests. DataDome behavior is server-side; no client version changes the outcome.

Steps

1. Confirm the DataDome interstitial and stop retrying into it

curl -s -o dd.html --max-time 25 -A "IntelBriefingBot/1.0" "https://YOUR-competitor/pricing"
grep -il "datadome" dd.html && echo "datadome interstitial confirmed"

Expected: Confirmation of the interstitial. Retrying the same request harder only burns time; the challenge does not resolve for scripted clients no matter the timeout.

2. Check the site's terms and robots before doing anything else

curl -s "https://YOUR-competitor/robots.txt" | grep -i pricing
curl -s "https://YOUR-competitor/terms" -o terms.html && grep -il "scrap\|crawl\|automat" terms.html

Expected: Whether the pricing path is disallowed and whether the terms forbid automated collection. If either says no, that settles it; do not look for a technical workaround.

3. Fetch the public pricing page politely, once, and cache it

import requests
s = requests.Session()
s.headers.update({"User-Agent": "IntelBriefingBot/1.0"})
r = s.get("https://YOUR-competitor/pricing", timeout=30)
print(r.status_code, len(r.text))
open("pricing_snapshot.html", "w").write(r.text)
print("cached one snapshot; re-fetch weekly at most")

Expected: One cached snapshot. If this single polite request is also challenged, the page is fully defended; stop fetching it entirely.

4. Re-source from a licensed or official channel

curl -s "https://YOUR-competitor/sitemap.xml" | grep -i "pricing\|plans" | head -10

Expected: Allowed pricing-related pages, or nothing. Pricing data vendors license this exact data with the vendor's cooperation; for briefing purposes their feed or the competitor's own announcements usually suffice.

Other ways people phrase this

datadome blocking scraper timeout fix

The timeout is the symptom, the challenge is the cause. Raising timeouts never resolves an interstitial; the fix is a different source.

competitor pricing page bot protection

Pricing pages are among the most defended pages on the web. Plan intel collection around that instead of treating each block as a surprise.

403 datadome cookie challenge loop

The cookie dance is the protection working. Scripted clients cannot complete it, which is the intended result.

Why it happens

DataDome fingerprints clients and serves bot-like traffic an interstitial challenge that only real browsers pass. Competitor pricing pages get this protection because they are scraped constantly and the data has commercial value. The timeout is not a performance bug; it is what an unpassable challenge looks like from the scraper's side.

Edge cases

• A single polite request that succeeds does not license polling; cache aggressively and re-fetch on a slow schedule.
• Pricing vendors that license data from the competitor directly are the clean path for systematic tracking.
• Press releases and earnings calls often disclose pricing changes with more context than the page itself.
• If your IP range is datacenter space, even polite requests score worse; that is the protection doing its job, not a misconfiguration.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstyLyxl7qtehp2rqD9vCaIg