scrapy-splash 502 bad gateway on render.html endpoint

TL;DR

The 502 means Splash (or the proxy in front of it) failed to produce a rendered page. Check that the Splash container is actually up and reachable from Scrapy, then look at render timeouts and page weight. Most 502s are Splash being down, overloaded, or killed by Docker networking, not your spider.

scrapy-splash 502 bad gateway on render.html endpoint

Use this when

• render.html returned 200 before and now returns 502.
• The Splash container logs show restarts or OOM kills.
• Scrapy and Splash run in separate containers or hosts.

Not for this skill when

• Splash returns 200 but the page content is wrong. That is a rendering or Lua problem.
• The 502 comes from the target site directly. Check the URL you are rendering.

Steps

1. Check the Splash container: docker ps shows it running, and its logs show no recent crashes. Verify: uptime is continuous and memory is not maxed.
2. Curl the endpoint directly from the Scrapy host: curl "http://SPLASH_HOST:8050/render.html?url=https://example.com". Verify: curl gets a 200, proving the network path works.
3. If curl 502s too, restart Splash and watch memory. Splash leaks on heavy JS pages; a scheduled restart is a legitimate fix. Verify: post-restart renders succeed.
4. Raise the render timeout for heavy pages and lower concurrency against Splash. Verify: the 502 rate drops on JS-heavy targets.
5. If Splash is up, reachable, and still 502s on specific URLs, those pages are killing the renderer (huge DOM, infinite scroll). Render with a Lua script that waits and screenshots early, or drop JS rendering for those URLs. Verify: the specific URLs render or are logged as skipped.

Variant phrasings

splash 502 bad gateway

The short search, same service-health diagnosis.

scrapy splash render.html error

Endpoint-named phrasing, points at step 2.

splash docker 502

Deployment phrasing, lands on the container check.

Compatibility: scrapy-splash with Splash 3.x in Docker. The 502 is almost always the service, not the spider.

Why it happens

render.html is a thin endpoint over a full browser engine. When the engine crashes, runs out of memory, or cannot reach the network, the HTTP layer has nothing to return and answers 502. Docker networking adds a second failure mode: Scrapy cannot reach Splash at all, and whatever proxy sits between them returns the 502.

Edge cases / pitfalls

• Splash's default memory limits are small for modern JS pages. Raise the container memory before tuning anything else.
• A 502 on every URL right after deploy is a networking problem, not a load problem. Check container DNS and port exposure first.
• Lua script errors surface as 400s or error JSON, not 502s. Do not debug scripts for a 502.
• If you outgrow Splash, Playwright with persistent contexts is the modern replacement, not a bigger Splash box.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_avKV4i2rs7KXwK1skJZbuQ