RangeError: Map maximum size exceeded when uploading a large file to the key-value store

Do not hand a giant Buffer to Actor.setValue(). Stream it instead: convert the data into a Readable stream and pass that to Actor.setValue() - the reporter confirmed this works for almost any file size. The maintainer also pointed out that building massive arrays or buffers is slow even without Crawlee in the mix, so if creating the buffer itself hangs, profile that step on its own before blaming the SDK.

Context: Issue apify/crawlee#1732 (closed, 38 comments): uploading a zip file over 9 MB to an Actor's key-value store with Actor.setValue() crashed with RangeError - Map maximum size exceeded. Creating the zip took 5 seconds, but saving it to the key-value store on the reporter's machine crashed after 15 minutes of runtime.

Matched source

Source: Published skill Original query: "RangeError: Map maximum size exceeded when uploading a large file to the key-value store" Key terms: exceeded, file, large, maximum, rangeerror, size, store, uploading, value

Source: Issue apify/crawlee#1732 (closed, 38 comments): uploading a zip file over 9 MB to an Actor's key-value store with Actor.setValue() crashed with RangeError - Map maximum size exceeded. Creating the zip took 5 seconds, but saving it to the key-value store on the reporter's machine crashed after 15 minutes of runtime.

Verified resolution: Do not hand a giant Buffer to Actor.setValue(). Stream it instead: convert the data into a Readable stream and pass that to Actor.setValue() - the reporter confirmed this works for almost any file size. The maintainer also pointed out that building massive arrays or buffers is slow even without Crawlee in the mix, so if creating the buffer itself hangs, profile that step on its own before blaming the SDK.

Source URL: https://github.com/apify/crawlee/issues/1732