Skip to content
← Back to activity

Immutable solution version

How to pass data between handlers in Crawlee Python, from the listing-page handler to the detail-page handler

Version 1Sign in to save

Complete solution

Maintainers answered: pass a user_data dict on enqueue_links (or, for per-link data, build Request.from_url(link, user_data={...}) and send it via context.add_requests), then read it in the other handler via context.request.user_data. One gotcha the reporter hit and confirmed: user_data must be JSON-serializable, because request queues serialize to disk/JSON, so values like datetimes silently come back as strings.

How to pass data between handlers in Crawlee Python, from the listing-page handler to the detail-page handler

What is going on

They were scraping half of each item's data in the listing-page handler and the other half in the detail-page handler, and needed to carry data (like a screening start time scraped on the listing page) down into the detail handler to combine with the detail data. They could not see how to pass information from one request to another between handlers, since the detail page is enqueued as a separate request.

What actually works

Maintainers answered: pass a userdata dict on enqueuelinks (or, for per-link data, build Request.fromurl(link, userdata={...}) and send it via context.addrequests), then read it in the other handler via context.request.userdata. One gotcha the reporter hit and confirmed: userdata must be JSON-serializable, because request queues serialize to disk/JSON, so values like datetimes silently come back as strings. Serialize complex types first (the reporter used pydantic modeldumpjson / modelvalidate_json) and the data arrives intact.

Source: https://github.com/apify/crawlee-python/issues/524

Source

Mined from a verified thread: https://vectle.com/threads/thr_1BWNGHvnQ5z-ZjTLz8eVxg (original: https://github.com/apify/crawlee-python/issues/524)

Evidence

In the thread, the asker reported: They were scraping half of each item's data in the listing-page handler and the other half in the detail-page handler, and needed to carry data (like a screening start time scraped on the listing page) down into the detail handler to combine with the detail data. They could not see how to pass information from one request to another between handlers, since the detail page is enqueued as a separate request. The verified answer that resolved it: Maintainers answered: pass a userdata dict on enqueuelinks (or, for per-link data, build Request.fromurl(link, userdata={...}) and send it via context.addrequests), then read it in the other handler via context.request.userdata. One gotcha the reporter hit and confirmed: userdata must be JSON-serializable, because request queues serialize to disk/JSON, so values like datetimes silently come back as strings. Serialize complex types first (the reporter used pydantic modeldumpjson / modelvalidate_json) and the data arrives intact.

Source: https://github.com/apify/crawlee-python/issues/524 Thread: https://vectle.com/threads/thr_1BWNGHvnQ5z-ZjTLz8eVxg

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_hmIeP5Ggykso0hf6fLDwHw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">How to pass data between handlers in Crawlee Python, from the listing-page handler to the detail-page handler <span>— Vectle</span></a>