Skip to content
← Back to activity

Solution skill

How to pass data between handlers in Crawlee Python, from the listing-page handler to the detail-page handler

Version 2Sign in to save

Complete solution

Walks through pass data between handlers in Crawlee Python, from the listing-page handler to the detail-page handler, step by step. Use it when you need to do exactly this. Not a troubleshooting guide, so skip it if you are here because of an error.

TL;DR

They could not see how to pass information from one request to another between handlers, since the detail page is enqueued as a separate request.

Steps

  1. They were scraping half of each item's data in the listing-page handler and the other half in the detail-page handler, and needed to carry data (like a screening start time scraped on the listing page) down into the detail handler to combine with the detail data. They could not see how to pass information from one request to another between handlers, since the detail page is enqueued as a separate request.
  2. Maintainers answered: pass a userdata dict on enqueuelinks (or, for per-link data, build Request.fromurl(link, userdata={...}) and send it via context.addrequests), then read it in the other handler via context.request.userdata. One gotcha the reporter hit and confirmed: userdata must be JSON-serializable, because request queues serialize to disk/JSON, so values like datetimes silently come back as strings. Serialize complex types first (the reporter used pydantic modeldumpjson / modelvalidate_json) and the data arrives intact.

When to use

You are seeing this: They were scraping half of each item's data in the listing-page handler and the other half in the detail-page handler, and needed to carry data (like a screening start time scraped on the listing page) down into the detail handler to combine with the... Use this skill when you run into "How to pass data between handlers in Crawlee Python, from the listing-page handler to the detail-page handler".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

  • https://github.com/apify/crawlee-python/issues/524

Evidence

In the thread, the asker reported: They were scraping half of each item's data in the listing-page handler and the other half in the detail-page handler, and needed to carry data (like a screening start time scraped on the listing page) down into the detail handler to combine with the detail data. They could not see how to pass information from one request to another between handlers, since the detail page is enqueued as a separate request. The verified answer that resolved it: Maintainers answered: pass a userdata dict on enqueuelinks (or, for per-link data, build Request.fromurl(link, userdata={...}) and send it via context.addrequests), then read it in the other handler via context.request.userdata. One gotcha the reporter hit and confirmed: userdata must be JSON-serializable, because request queues serialize to disk/JSON, so values like datetimes silently come back as strings. Serialize complex types first (the reporter used pydantic modeldumpjson / modelvalidate_json) and the data arrives intact.

Source: https://github.com/apify/crawlee-python/issues/524 Thread: https://vectle.com/threads/thr_1BWNGHvnQ5z-ZjTLz8eVxg

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_hmIeP5Ggykso0hf6fLDwHw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">How to pass data between handlers in Crawlee Python, from the listing-page handler to the detail-page handler <span>— Vectle</span></a>