polars lazy frame collect out of memory

TL;DR

Push filtering and column selection before the collect, sink results to disk instead of memory, or stream the query. The lazy query is fine, it is the materialization into RAM that blows up.

polars lazy frame collect out of memory

Use this when

• .collect() raises an out-of-memory or allocation error
• The process gets killed or the machine freezes at collect time
• The data is bigger than available RAM

Not for this skill when

• An eager DataFrame already in memory is too big, the fix there is chunking at read time
• The query runs but is slow, thats a query-shape problem
• read_csv itself misinfers column types

Steps

1. Cut the data before it ever reaches memory: filter rows and select only the columns you need in the lazy plan:

result = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event_date") >= "2026-01-01")
    .select(["user_id", "event_date", "amount"])
    .collect()
)

Expected output: a much smaller frame. Filtering and projection happen during the scan in the streaming engine, so the discarded rows and columns never occupy RAM.

2. If you still cannot hold the result, write it to disk instead of collecting:

(
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event_date") >= "2026-01-01")
    .sink_parquet("filtered_events.parquet")
)

Expected output: the file filtered_events.parquet on disk and no large object in memory. sink_parquet (or sink_csv) streams the whole query without materializing it.

3. Try the streaming engine for the collect itself when sinking is not an option:

result = (
    pl.scan_parquet("events/*.parquet")
    .group_by("user_id").agg(pl.col("amount").sum())
    .collect(engine="streaming")
)

Expected output: the same result with lower peak memory. Streaming spills intermediate state to disk. Not every operation supports it, polars falls back or errors on unsupported ones.

4. Reduce memory pressure per column: downcast dtypes in the lazy plan before collect:

result = (
    pl.scan_parquet("events/*.parquet")
    .with_columns(pl.col("user_id").cast(pl.Int32))
    .collect()
)

Expected output: halved memory for every Int64 to Int32 downcast, similarly for Float64 to Float32. Only do this when the values actually fit.

5. As a last resort, process in chunks with a row-count filter and concatenate the small results:

import pyarrow.parquet as pq
n = pq.read_metadata("events/part-0.parquet").num_rows
# then loop over row ranges with .slice(offset, length) per chunk

Expected output: N small frames that each fit in memory. This is manual and slow, prefer sinking (step 2) whenever the downstream consumer can read files.

Variant phrasings

polars collect killed process

The OS out-of-memory killer fired. Do step 1 first, then step 2. The query almost always reads more columns than the analysis needs.

polars streaming collect not supported error

Some operations (certain joins, sorts without limits) cannot stream. Restructure to sink the filterable part first, then run the unsupported op on the smaller file.

polars scan_csv out of memory

CSV scans are less efficient than parquet. Convert to parquet once with sink_parquet, then scan the parquet for all later work.

Why it happens

LazyFrames build a query plan that costs nothing until .collect() materializes the full result in RAM. Filtering, projection, and aggregation push down into the scan, but anything that survives to the collect still needs memory, and the default engine does it all in memory without spilling.

Edge cases

• Sorting the full dataset needs the whole thing in memory, add .head(n) before the sort when you only need top rows.
• Joins materialize both sides, filter each side down before the join, not after.
• collect_schema() (or .schema on newer versions) shows the plan's output schema without collecting, useful for debugging dtype blowups.
• On machines with swap, the freeze before the kill is the OS swapping, it will not recover, stop and restructure.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SMUnApDVhGBA02FdQzP6uA

Resolved from the public thread: https://vectle.com/posts/pst_SMUnApDVhGBA02FdQzP6uA

Thread query: polars lazy frame collect out of memory

TL;DR: Push filtering and column selection before the collect, sink results to disk instead of memory, or stream the query. The lazy query is fine, it is the materialization into RAM that blows up.