Immutable solution version
Each and every update to crawlee breaks Dataset.pushData on Apify Platform
Complete solution
Shows how to fix each and every update to crawlee breaks Dataset.pushData on Apify Platform. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.
TL;DR
Use pushData from the crawling context, do not open the request queue explicitly when you do not need to, and pass config: Actor.config so everything shares one config. Source: https://github.com/apify/crawlee/issues/2117 (Crawlee)
Steps
- If Dataset.pushData misbehaves after a Crawlee update, check for duplicate Crawlee installs: multiple copies each create their own global storage config, which is what breaks pushData on the Apify platform. Use pushData from the crawling context, do not open the request queue explicitly when you do not need to, and pass config: Actor.config so everything shares one config.
- Source: https://github.com/apify/crawlee/issues/2117 (Crawlee)
When to use
Use this skill when you run into "Each and every update to crawlee breaks Dataset.pushData on Apify Platform".
When not to use
If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.
Versions
No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.
Why this happens
The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.
Sources
- https://github.com/apify/crawlee/issues/2117
Evidence
In gh:apify/crawlee#2117, the issue was: ### Which package is this bug report for? If unsure which one to select, leave blank @crawlee/core ### Issue description Whenever crawlee is bumped, all the Dataset.pushData functionalities get broken and without any error, they just don't proceed with any. The package.json is the following `` "dependencies": { "apify": "^3.1.11", "cheerio": "1.0.0-rc.12", "crawlee": "^3.5.7", "got-scraping": "3.2.15" }, ### Code sample ``TypeScript import { BasicCrawler, log, Dataset } from 'crawlee'; await Dataset.pushData({a: 1});. The verified answer: If Dataset.pushData misbehaves after a Crawlee update, check for duplicate Crawlee installs: multiple copies each create their own global storage config, which is what breaks pushData on the Apify platform. Use pushData from the crawling context, do not open the request queue explicitly when you do not need to, and pass config: Actor.config so everything shares one config.. Issue URL: https://github.com/apify/crawlee/issues/2117
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_Thx5UCTWAyl8XbA6-CcDBA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Each and every update to crawlee breaks Dataset.pushData on Apify Platform <span>— Vectle</span></a>