twisted.internet.defer.TimeoutError scrapy

TL;DR

This timeout means a single Scrapy request ran longer than DOWNLOADTIMEOUT (180 seconds by default). Raise DOWNLOADTIMEOUT for slow sites, or lower concurrency and add a download delay so the target keeps up.

twisted.internet.defer.TimeoutError scrapy

Use this when

• Spiders die mid-crawl with a twisted TimeoutError.
• Large file downloads or slow APIs time out.
• The same spider worked on a fast network but fails on a slow one.

Not for this skill when

• The error is DNS or connection refused. Those fail fast and are not timeouts.
• You need pages faster, not slower. Timeouts mean slow down, not speed up.

Steps

1. Confirm the failing URLs are slow: fetch one with curl and time it. Verify: you know the real response time.
2. Set DOWNLOAD_TIMEOUT = 300 in settings.py (or higher than the slowest page). Verify: the setting appears in the spider's logged settings dump.
3. If the server is slow for everyone, add DOWNLOAD_DELAY = 1 and drop CONCURRENT_REQUESTS to 8. Verify: the TimeoutErrors stop and throughput stays steady.
4. For one-off slow endpoints, set the timeout per request: request.meta['download_timeout'] = 600. Verify: only that request gets the longer window.
5. Re-run the crawl. Verify: zero TimeoutErrors in the log and the item count matches expectations.

Variant phrasings

scrapy download timeout

Same setting, general search.

twisted timeouterror spider

Covers the traceback search.

scrapy request times out

Symptom-level search for the same failure.

Compatibility: Scrapy 2.x. DOWNLOAD_TIMEOUT is in seconds; the default is 180.

Why it happens

Twisted fires this when the server takes longer than DOWNLOAD_TIMEOUT to send the full response. Common causes: slow APIs, large payloads, overloaded servers, or a spider hammering a small site so hard that every response queues behind the others.

Edge cases / pitfalls

• Raising the timeout on a dead server just waits longer to fail. Confirm the URL responds at all before raising it.
• Very high timeouts with high concurrency can pin all slots on hung requests. Raise the timeout and lower concurrency together.
• Retries on timed-out requests double the load. Keep RETRY_TIMES modest when the server is the bottleneck.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DKsy3lZuM2gB7-cDciRV-A