## TL;DR
This timeout means a single Scrapy request ran longer than DOWNLOAD_TIMEOUT (180 seconds by default). Raise DOWNLOAD_TIMEOUT for slow sites, or lower concurrency and add a download delay so the target keeps up.

```text
twisted.internet.defer.TimeoutError scrapy
```

## Use this when
- Spiders die mid-crawl with a twisted TimeoutError.
- Large file downloads or slow APIs time out.
- The same spider worked on a fast network but fails on a slow one.

## Not for this skill when
- The error is DNS or connection refused. Those fail fast and are not timeouts.
- You need pages faster, not slower. Timeouts mean slow down, not speed up.

## Steps
1. Confirm the failing URLs are slow: fetch one with curl and time it. Verify: you know the real response time.
2. Set `DOWNLOAD_TIMEOUT = 300` in settings.py (or higher than the slowest page). Verify: the setting appears in the spider's logged settings dump.
3. If the server is slow for everyone, add `DOWNLOAD_DELAY = 1` and drop `CONCURRENT_REQUESTS` to 8. Verify: the TimeoutErrors stop and throughput stays steady.
4. For one-off slow endpoints, set the timeout per request: `request.meta['download_timeout'] = 600`. Verify: only that request gets the longer window.
5. Re-run the crawl. Verify: zero TimeoutErrors in the log and the item count matches expectations.

## Variant phrasings
### scrapy download timeout
Same setting, general search.
### twisted timeouterror spider
Covers the traceback search.
### scrapy request times out
Symptom-level search for the same failure.

Compatibility: Scrapy 2.x. DOWNLOAD_TIMEOUT is in seconds; the default is 180.

## Why it happens
Twisted fires this when the server takes longer than DOWNLOAD_TIMEOUT to send the full response. Common causes: slow APIs, large payloads, overloaded servers, or a spider hammering a small site so hard that every response queues behind the others.

## Edge cases / pitfalls
- Raising the timeout on a dead server just waits longer to fail. Confirm the URL responds at all before raising it.
- Very high timeouts with high concurrency can pin all slots on hung requests. Raise the timeout and lower concurrency together.
- Retries on timed-out requests double the load. Keep RETRY_TIMES modest when the server is the bottleneck.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DKsy3lZuM2gB7-cDciRV-A