Skip to content
← Back to activity

Immutable solution version

Each and every update to crawlee breaks Dataset.pushData on Apify Platform

Version 1Sign in to save

Complete solution

If Dataset.pushData misbehaves after a Crawlee update, check for duplicate Crawlee installs: multiple copies each create their own global storage config, which is what breaks pushData on the Apify platform. Use pushData from the crawling context, do not open the request queue explicitly when you do not need to, and pass config: Actor.config so everything shares one config.

If Dataset.pushData misbehaves after a Crawlee update, check for duplicate Crawlee installs: multiple copies each create their own global storage config, which is what breaks pushData on the Apify platform. Use pushData from the crawling context, do not open the request queue explicitly when you do not need to, and pass config: Actor.config so everything shares one config.

Source: gh:apify/crawlee#2117 (Crawlee)

Evidence

In gh:apify/crawlee#2117, the issue was: ### Which package is this bug report for? If unsure which one to select, leave blank @crawlee/core ### Issue description Whenever crawlee is bumped, all the Dataset.pushData functionalities get broken and without any error, they just don't proceed with any. The package.json is the following `` "dependencies": { "apify": "^3.1.11", "cheerio": "1.0.0-rc.12", "crawlee": "^3.5.7", "got-scraping": "3.2.15" }, ### Code sample ``TypeScript import { BasicCrawler, log, Dataset } from 'crawlee'; await Dataset.pushData({a: 1});. The verified answer: If Dataset.pushData misbehaves after a Crawlee update, check for duplicate Crawlee installs: multiple copies each create their own global storage config, which is what breaks pushData on the Apify platform. Use pushData from the crawling context, do not open the request queue explicitly when you do not need to, and pass config: Actor.config so everything shares one config.. Issue URL: https://github.com/apify/crawlee/issues/2117

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_Thx5UCTWAyl8XbA6-CcDBA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Each and every update to crawlee breaks Dataset.pushData on Apify Platform <span>— Vectle</span></a>