library · inferred from evidence
Apify/Crawlee
A web scraping and automation library for Node.js, referred to as Crawlee in the context of Apify.
- Use priority queue by number of url finds
Wontfix, by design. Crawlee will not get a priority queue that orders URLs by how many other pages link to them: it would have to be implemented across the Apify platform and API, not just in Crawlee, and URL frequency is not stored anywher
- PlaywrightCrawler hangs infinitely when closing page
[barjin (maintainer)]: The crawler hang when a page refuses to close was fixed by PRs 4108 and 4116: the browser pool now handles page-close timeouts itself and retires the browser when a close never settles, instead of holding a live brows
- Each and every update to crawlee breaks Dataset.pushData on Apify Platform
If Dataset.pushData misbehaves after a Crawlee update, check for duplicate Crawlee installs: multiple copies each create their own global storage config, which is what breaks pushData on the Apify platform. Use pushData from the crawling co
- CheerioCrawler overwrites request cookies when persistCookiesPerSession is false
How to fix: CheerioCrawler overwrites request cookies when persistCookiesPerSession is false. The verified fix from the maintainer thread, distilled into reusable steps.
- Crawlee SQLite "disk I/O error" in a Cloud Run container
A Crawlee crawler works locally but throws SQLite disk I/O error at RequestQueue creation on Cloud Run because WAL mode does not work on its filesystem. Disable WAL with APIFY_LOCAL_STORAGE_ENABLE_WAL_MODE=false.