## TL;DR
Push filtering and column selection before the collect, sink results to disk instead of memory, or stream the query. The lazy query is fine, it is the materialization into RAM that blows up.

```text
polars lazy frame collect out of memory
```

## Use this when
- `.collect()` raises an out-of-memory or allocation error
- The process gets killed or the machine freezes at collect time
- The data is bigger than available RAM

## Not for this skill when
- An eager DataFrame already in memory is too big, the fix there is chunking at read time
- The query runs but is slow, thats a query-shape problem
- `read_csv` itself misinfers column types

## Steps

1. Cut the data before it ever reaches memory: filter rows and select only the columns you need in the lazy plan:

```python
result = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event_date") >= "2026-01-01")
    .select(["user_id", "event_date", "amount"])
    .collect()
)
```
Expected output: a much smaller frame. Filtering and projection happen during the scan in the streaming engine, so the discarded rows and columns never occupy RAM.

2. If you still cannot hold the result, write it to disk instead of collecting:

```python
(
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event_date") >= "2026-01-01")
    .sink_parquet("filtered_events.parquet")
)
```
Expected output: the file `filtered_events.parquet` on disk and no large object in memory. `sink_parquet` (or `sink_csv`) streams the whole query without materializing it.

3. Try the streaming engine for the collect itself when sinking is not an option:

```python
result = (
    pl.scan_parquet("events/*.parquet")
    .group_by("user_id").agg(pl.col("amount").sum())
    .collect(engine="streaming")
)
```
Expected output: the same result with lower peak memory. Streaming spills intermediate state to disk. Not every operation supports it, polars falls back or errors on unsupported ones.

4. Reduce memory pressure per column: downcast dtypes in the lazy plan before collect:

```python
result = (
    pl.scan_parquet("events/*.parquet")
    .with_columns(pl.col("user_id").cast(pl.Int32))
    .collect()
)
```
Expected output: halved memory for every Int64 to Int32 downcast, similarly for Float64 to Float32. Only do this when the values actually fit.

5. As a last resort, process in chunks with a row-count filter and concatenate the small results:

```python
import pyarrow.parquet as pq
n = pq.read_metadata("events/part-0.parquet").num_rows
# then loop over row ranges with .slice(offset, length) per chunk
```
Expected output: N small frames that each fit in memory. This is manual and slow, prefer sinking (step 2) whenever the downstream consumer can read files.

## Variant phrasings

### polars collect killed process
The OS out-of-memory killer fired. Do step 1 first, then step 2. The query almost always reads more columns than the analysis needs.

### polars streaming collect not supported error
Some operations (certain joins, sorts without limits) cannot stream. Restructure to sink the filterable part first, then run the unsupported op on the smaller file.

### polars scan_csv out of memory
CSV scans are less efficient than parquet. Convert to parquet once with `sink_parquet`, then scan the parquet for all later work.

## Why it happens
LazyFrames build a query plan that costs nothing until `.collect()` materializes the full result in RAM. Filtering, projection, and aggregation push down into the scan, but anything that survives to the collect still needs memory, and the default engine does it all in memory without spilling.

## Edge cases
- Sorting the full dataset needs the whole thing in memory, add `.head(n)` before the sort when you only need top rows.
- Joins materialize both sides, filter each side down before the join, not after.
- `collect_schema()` (or `.schema` on newer versions) shows the plan's output schema without collecting, useful for debugging dtype blowups.
- On machines with swap, the freeze before the kill is the OS swapping, it will not recover, stop and restructure.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SMUnApDVhGBA02FdQzP6uA