RangeError: Map maximum size exceeded when uploading large file to Apify key-value store

RangeError: Map maximum size exceeded when uploading large file to Apify key-value store

What is going on

Uploading a zip file larger than about 9 MB to an Apify Actor's key-value store with Actor.setValue() (crawlee 3.1.3 / apify 3.1.1) throws RangeError: Map maximum size exceeded. Building the archive itself takes only ~5 seconds; the crash happens while saving to the key-value store. Smaller files save without any issue, so the failure is size-dependent.

What actually works

Pass the data as a ReadableStream instead of a fully materialized Buffer/string. Convert the file into a ReadableStream and hand that to Actor.setValue(); the reporter confirmed this works for almost any file size. The crash comes from the store materializing the whole payload in a Map-backed structure, which a stream avoids. Source: https://github.com/apify/crawlee/issues/1732

Source

Mined from a verified thread: https://vectle.com/threads/thr__7MxyYvptVhbwxuoVacimg (original: https://github.com/apify/crawlee/issues/1732)

In the thread, the asker reported: Uploading a zip file larger than about 9 MB to an Apify Actor's key-value store with Actor.setValue() (crawlee 3.1.3 / apify 3.1.1) throws RangeError: Map maximum size exceeded. Building the archive itself takes only ~5 seconds; the crash happens while saving to the key-value store. Smaller files save without any issue, so the failure is size-dependent. The verified answer that resolved it: Pass the data as a ReadableStream instead of a fully materialized Buffer/string. Convert the file into a ReadableStream and hand that to Actor.setValue(); the reporter confirmed this works for almost any file size. The crash comes from the store materializing the whole payload in a Map-backed structure, which a stream avoids. Source: https://github.com/apify/crawlee/issues/1732 Thread: https://vectle.com/threads/thr__7MxyYvptVhbwxuoVacimg