Immutable solution version
How to pass data between handlers in Crawlee Python, from the listing-page handler to the detail-page handler
Complete solution
Maintainers answered: pass a user_data dict on enqueue_links (or, for per-link data, build Request.from_url(link, user_data={...}) and send it via context.add_requests), then read it in the other handler via context.request.user_data. One gotcha the reporter hit and confirmed: user_data must be JSON-serializable, because request queues serialize to disk/JSON, so values like datetimes silently come back as strings.
How to pass data between handlers in Crawlee Python, from the listing-page handler to the detail-page handler
What is going on
They were scraping half of each item's data in the listing-page handler and the other half in the detail-page handler, and needed to carry data (like a screening start time scraped on the listing page) down into the detail handler to combine with the detail data. They could not see how to pass information from one request to another between handlers, since the detail page is enqueued as a separate request.
What actually works
Maintainers answered: pass a userdata dict on enqueuelinks (or, for per-link data, build Request.fromurl(link, userdata={...}) and send it via context.addrequests), then read it in the other handler via context.request.userdata. One gotcha the reporter hit and confirmed: userdata must be JSON-serializable, because request queues serialize to disk/JSON, so values like datetimes silently come back as strings. Serialize complex types first (the reporter used pydantic modeldumpjson / modelvalidate_json) and the data arrives intact.
Source: https://github.com/apify/crawlee-python/issues/524
Source
Mined from a verified thread: https://vectle.com/threads/thr_1BWNGHvnQ5z-ZjTLz8eVxg (original: https://github.com/apify/crawlee-python/issues/524)
Evidence
In the thread, the asker reported: They were scraping half of each item's data in the listing-page handler and the other half in the detail-page handler, and needed to carry data (like a screening start time scraped on the listing page) down into the detail handler to combine with the detail data. They could not see how to pass information from one request to another between handlers, since the detail page is enqueued as a separate request. The verified answer that resolved it: Maintainers answered: pass a userdata dict on enqueuelinks (or, for per-link data, build Request.fromurl(link, userdata={...}) and send it via context.addrequests), then read it in the other handler via context.request.userdata. One gotcha the reporter hit and confirmed: userdata must be JSON-serializable, because request queues serialize to disk/JSON, so values like datetimes silently come back as strings. Serialize complex types first (the reporter used pydantic modeldumpjson / modelvalidate_json) and the data arrives intact.
Source: https://github.com/apify/crawlee-python/issues/524 Thread: https://vectle.com/threads/thr_1BWNGHvnQ5z-ZjTLz8eVxg
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_hmIeP5Ggykso0hf6fLDwHw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">How to pass data between handlers in Crawlee Python, from the listing-page handler to the detail-page handler <span>— Vectle</span></a>