Typesense OOM crash when exporting a large collection

TL;DR

Upgrade to 0.25.0 or later to export large collections without running the server out of memory. The v0.24.1 patch notes claimed an improved export memory footprint, but the export failed the same way on both v0.24.0 and v0.24.1.

Steps

1. Exporting a large collection (~4.5M documents) via the collections/[name]/export endpoint maxed out memory and crashed typesense-server fatally. The v0.24.1 patch notes claimed an improved export memory footprint, but the export failed the same way on both v0.24.0 and v0.24.1. With ~3M records the same export had worked, so the reporter hit a hard scaling wall.
2. The export memory blowup was comprehensively fixed in Typesense 0.25.0. Upgrade to 0.25.0 or later to export large collections without running the server out of memory.

When to use

You are seeing this: Exporting a large collection (~4.5M documents) via the collections/[name]/export endpoint maxed out memory and crashed typesense-server fatally. Use this skill when you run into "Typesense OOM crash when exporting a large collection".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: v0.24.1, both v0.24.0, Typesense 0.25.0. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Asked in the thread (https://vectle.com/threads/thrkzOmkxpJLIq81juXP44ntQ): Exporting a large collection (~4.5M documents) via the collections/[name]/export endpoint maxed out memory and crashed typesense-server fatally. The v0.24.1 patch notes claimed an improved export memory footprint, but the export failed the same way on both v0.24.0 and v0.24.1. With ~3M records the same export had worked, so the reporter hit a hard scaling wall. The verified answer: The export memory blowup was comprehensively fixed in Typesense 0.25.0. Upgrade to 0.25.0 or later to export large collections without running the server out of memory. Source: https://github.com/typesense/typesense/issues/1067 Thread: https://vectle.com/threads/thrkzOmkxpJLIq81juXP44ntQ