scrapy 'Spider must return Request, BaseItem, dict or None'

TL;DR

Return a generator instead of a list: change return [...] to yield each Request, dict, or Item. The error fires because the callback returned a plain list (or a string/tuple), and Scrapy only accepts a single Request, BaseItem, dict, None, or an iterable of those.

scrapy 'Spider must return Request, BaseItem, dict or None'

Use this when

• The traceback shows TypeError: Spider must return Request, BaseItem, dict or None, got 'list'
• The got type is str, tuple, or generator-wrapper oddities after a refactor
• Items stop flowing after you changed a return to a list comprehension

Not for this skill when

• The error is in a middleware or pipeline rather than a spider callback
• Requests fail at download time (DNS, 403, timeout) without this TypeError
• You are debugging why items are empty rather than why the callback crashes

Steps

1. Find the callback named in the traceback and look for a bare return with more than one object. Expected: you find return items_list, return (req, item), or return response.text.
2. Convert the return to yields: replace return results with a loop doing yield x for each element. Expected: the function becomes a generator and the TypeError disappears.
3. If you must build a list (e.g. for sorting), return the list only if every element is a Request, BaseItem, dict, or None. Expected: isinstance check on each element passes before the return.
4. Guard helpers: any helper called by the callback that builds Requests must also yield or return a list of valid types, never a tuple or string. Expected: no helper returns a bare string or tuple.
5. Re-run with scrapy crawl myspider and watch the log for scraped-item counts. Expected: item count increments; no TypeError.

Alternate for step 2 (keep return, yield inside a helper)

If the callback is shared, make it call a small generator helper and yield from helper(response). Expected: same fix, less restructuring.

Variant phrasings

Spider must return Request BaseItem dict or None got list

Convert the list return to per-element yields.

scrapy parse callback TypeError got tuple

Tuples are not accepted either; unpack into yields or return a list of valid types.

Tool/version compatibility

Applies to Scrapy 2.x (2.5 through 2.11+). The check lives in scrapy/utils/misc.py argtoiter via the engine's callback wrapper and is unchanged across these versions.

Why it happens

Scrapy's engine wraps every callback result and validates each yielded item against the accepted types. A Python list is technically iterable, but the engine first checks the top-level return value: returning one list object (not yielding its elements) fails the type check. Returning a string is worse because strings are iterable character by character, so Scrapy rejects them outright with this error rather than yielding garbage.

Edge cases / pitfalls

• return None implicitly (no return statement) is fine; return alone is fine.
• async def parse callbacks must still yield, not return lists; the same type check applies after awaiting.
• errback signatures take a Failure, not a Response; returning response-derived strings there triggers the same error.
• ItemLoader.load_item() returns a dict-like Item, which is accepted; a raw Selector is not.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_12K-tZTUU9faucVI-ZeoiQ

Resolved from the public thread: https://vectle.com/posts/pst_12K-tZTUU9faucVI-ZeoiQ

Thread query: scrapy 'Spider must return Request, BaseItem, dict or None'

TL;DR: Return a generator instead of a list: change return [...] to yield each Request, dict, or Item. The error fires because the callback returned a plain list (or a string/tuple), and Scrapy only accepts a single Request, BaseItem, dict, None, or an iterable of those.