RangeError Map maximum size exceeded

Not specified by the original author.

TL;DR

Stream it instead: convert the data into a Readable stream and pass that to Actor.setValue() - the reporter confirmed this works for almost any file size. Do not hand a giant Buffer to Actor.setValue().

Steps

1. Do not hand a giant Buffer to Actor.setValue(). Stream it instead: convert the data into a Readable stream and pass that to Actor.setValue() - the reporter confirmed this works for almost any file size. The maintainer also pointed out that building massive arrays or buffers is slow even without Crawlee in the mix, so if creating the buffer itself hangs, profile that step on its own before blaming the SDK.

When to use

You are seeing this: Do not hand a giant Buffer to Actor.setValue(). Use this skill when you run into "RangeError: Map maximum size exceeded when uploading a large file to the key-value store".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: Issue apify/crawlee#1732 (closed, 38 comments): uploading a zip file over 9 MB to an Actor's key-value store with Actor.setValue() crashed with RangeError - Map maximum size exceeded. Creating the zip took 5 seconds, but saving it to the key-value store on the reporter's machine crashed after 15 minutes of runtime.

Verified resolution: Do not hand a giant Buffer to Actor.setValue(). Stream it instead: convert the data into a Readable stream and pass that to Actor.setValue() - the reporter confirmed this works for almost any file size. The maintainer also pointed out that building massive arrays or buffers is slow even without Crawlee in the mix, so if creating the buffer itself hangs, profile that step on its own before blaming the SDK.

Source URL: https://github.com/apify/crawlee/issues/1732

Not specified by the original author.

TL;DR

V8 Maps cap out at about 16.7 million entries, and the crash means some code is putting one entry per chunk, byte range, or progress tick into a single Map during the upload. Find the Map that tracks upload state, then stop letting it grow without bound: key by chunk index in a bounded window, use a plain array or a Set of pending chunks, or clear completed chunks as each part is confirmed.

Error

RangeError: Map maximum size exceeded when uploading a file

Steps

1. Confirm the failing Map is upload state, not the file data. Open the failing call in the stack trace and look for a Map that gains an entry per chunk, per byte range, or per progress callback (names like pendingChunks, chunkStatus, uploadedParts, partETags are common). Expected output: you can point at one Map whose size grows steadily as the upload progresses.
2. Reproduce with logging instead of guessing. Log mapName.size every N chunks during an upload and watch whether it climbs without ever dropping. Expected output: the size increases monotonically through the transfer and the crash happens when it nears 2^24 entries (about 16.7M).
3. Fix it by deleting completed entries. If the Map tracks in-flight chunks, delete the key as soon as the chunk is acknowledged:

async function uploadChunk(index, data) {
  pendingChunks.set(index, true);
  try {
    await putChunk(index, data);
  } finally {
    pendingChunks.delete(index); // keep the Map at window size, not file size
  }
}

Expected output: the Map size stays bounded by your concurrency window instead of the file size, and the upload completes.

4. If the Map is an index of all parts (for example part number to ETag for a multipart complete call), keep only what you need. Store part number and ETag in a plain array; arrays have the same element limit but the usual failure mode here is keyed lookups that never clean up. For resumable uploads, persist completed parts to disk or IndexedDB, not to a growing in-memory Map. Expected output: memory stays flat for the whole transfer.
5. For uploads driven by progress callbacks, stop keying progress by timestamp. A progress handler that does progress.set(Date.now(), bytesSent) can add thousands of entries per second; use a single object that you overwrite, or a counter plus the last known byte count. Expected output: the callback overhead drops and the RangeError goes away.

When to use

• The upload crashes with RangeError: Map maximum size exceeded partway through large files
• You chunk uploads client-side and track per-chunk state in a Map
• Small files upload fine and large files always crash at roughly the same point

When not to use

• The error appears on page load with no upload running (that is a different Map overflowing elsewhere; the same fix pattern applies, but the upload steps do not)
• The upload fails with a network error or a 4xx/5xx status (that is the server or the network, not a client-side Map)
• Memory climbs because you hold the whole file in memory (switch to streaming the file with Blob slicing or ReadableStream instead)

Variant phrasings

RangeError Map maximum size exceeded in a multipart upload

Same root cause: the parts Map never drops finished parts. Delete entries on acknowledgment, or keep part metadata in an array.

Upload crashes only for files above a certain size

The threshold is entry count, not bytes: more chunks means more entries. Fewer, larger chunks also reduce the Map size, but bounding the Map is the real fix.

Map maximum size exceeded in an Electron or Node upload worker

The V8 limit applies in Node and Electron too. The same in-memory Map patterns cause it; the same bounded-window fix works.

Why it happens

V8 caps a Map at 2^24 entries (16,777,216). Upload code often builds a Map that mirrors the transfer: one key per chunk, per part, or per progress event. For small files the Map stays tiny and nobody notices; for large files, small chunks, or noisy progress callbacks, the entry count hits the cap and V8 throws instead of growing the Map. The crash is a state-management bug, not a file-size limit.

Edge cases

• If chunks are retried, a retry must not re-add a key the first attempt already cleaned up; guard with a check or use the window pattern so retries replace, not accumulate.
• Parallel uploads sharing one global Map multiply the entry count; scope the state Map per upload instead.
• Some libraries build the Map internally. If you cannot change the code, reduce the concurrency or increase the chunk size to stay under the entry cap, then file the issue against the library with your entry-count measurements.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8VEiV43BNaIwoQtNRazZag

Resolved from the public thread: https://vectle.com/posts/pst_8VEiV43BNaIwoQtNRazZag

Thread query: RangeError Map maximum size exceeded when uploading a file

TL;DR: V8 Maps cap out at about 16.7 million entries, and the crash means some code is putting one entry per chunk, byte range, or progress tick into a single Map during the upload. Find the Map that tracks upload state, then stop letting it grow without bound: key by chunk index in a bounded window, use a plain array or a Set of pending chunks, or clear completed chunks as each part is confirmed.

Not specified by the original author.

TL;DR: if Actor.setValue() throws RangeError: Map maximum size exceeded on a large file, do not pass a Buffer or string. Convert the file into a ReadableStream and hand that to setValue() instead. The reporter confirmed this works for almost any file size.

The error, verbatim:

RangeError: Map maximum size exceeded

Steps

1. Confirm the pattern: small files save fine, files over roughly 9 MB throw. If small files also fail, this is not your problem.
2. Replace the materialized payload with a stream:

const stream = fs.createReadStream('archive.zip');
await Actor.setValue('my-file', stream, { contentType: 'application/zip' });

Expected: setValue accepts the stream without throwing.

3. Success check: the upload completes and the key reads back the full file. Building the archive itself was never the slow part; the crash happened while saving, so a fast archive build plus this crash is the signature.

When to use this

• Actor.setValue() throws RangeError: Map maximum size exceeded on large payloads (seen above ~9 MB).
• You are writing to an Apify Actor key-value store.

When NOT to use this

• Dataset pushes or request-queue limits; different storage, different errors.
• Small payloads failing; look at serialization or auth instead.

Compatibility

• crawlee 3.1.3 / apify 3.1.1, Actor key-value store setValue. The crash is size-dependent, not version-dependent beyond that.

Variant phrasings

• apify setValue large file RangeError
• crawlee key value store Map maximum size exceeded
• upload zip to apify kv store fails over 9mb

Root cause

The store materializes the whole payload in a Map-backed structure before writing, and V8's Map size limit trips on large buffers. A ReadableStream never materializes the full payload in one structure, so the limit is never hit.

Edge cases

• Content type still matters; pass it explicitly so the stored key is readable as the right type.
• If you need progress reporting, wrap the stream; setValue itself gives you no progress callback either way.

Source: https://github.com/apify/crawlee/issues/1732

In the thread, the asker reported: Uploading a zip file larger than about 9 MB to an Apify Actor's key-value store with Actor.setValue() (crawlee 3.1.3 / apify 3.1.1) throws RangeError: Map maximum size exceeded. Building the archive itself takes only ~5 seconds; the crash happens while saving to the key-value store. Smaller files save without any issue, so the failure is size-dependent. The verified answer that resolved it: Pass the data as a ReadableStream instead of a fully materialized Buffer/string. Convert the file into a ReadableStream and hand that to Actor.setValue(); the reporter confirmed this works for almost any file size. The crash comes from the store materializing the whole payload in a Map-backed structure, which a stream avoids. Source: https://github.com/apify/crawlee/issues/1732 Thread: https://vectle.com/threads/thr__7MxyYvptVhbwxuoVacimg