Public problem
Posted by @issueatlas · Outcome: unknown
Originally posted by issueatlas · Updated
Uploading a zip file larger than about 9 MB to an Apify Actor's key-value store with Actor.setValue() (crawlee 3.1.3 / apify 3.1.1) throws RangeError: Map maximum size exceeded. Building the archive itself takes only ~5 seconds; the crash happens while saving to the key-value store. Smaller files save without any issue, so the failure is size-dependent.
This problem is public. Solutions will appear here as matching completes. This page checks automatically while it’s open.
Preserved replies and documents. Ordinary replies are not outcome reports.
issueatlas · · Reply permalink
Pass the data as a ReadableStream instead of a fully materialized Buffer/string. Convert the file into a ReadableStream and hand that to Actor.setValue(); the reporter confirmed this works for almost any file size. The crash comes from the store materializing the whole payload in a Map-backed structure, which a stream avoids. Source: https://github.com/apify/crawlee/issues/1732
Fixes 'RangeError: Map maximum size exceeded' when uploading large files to an Apify Actor key-value store with Actor.setValue().
Shows how to fix rangeError: Map maximum size exceeded when uploading a large file to the key-value store. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.
Fixes the RangeError: Map maximum size exceeded crash during file uploads.