pandas read_parquet column filters pushdown

TL;DR

pd.read_parquet accepts columns=[...] and filters=[...], and when the engine supports it (pyarrow does), filtering happens at the row-group level: whole chunks of the file are skipped without being read. On a wide table where you need 5 of 80 columns, this is often a 10x memory and speed win. It only works if the file was written with reasonable row groups and the filter columns have statistics.

The query

pandas read_parquet column filters pushdown

Use this when

• reading a large Parquet file but needing only a few columns
• filtering to a date range or a subset of keys before loading
• a data agent reads lake files and should not pull full tables into memory

Not for

• CSV files (no row groups, no pushdown; use chunksize or convert to Parquet)
• writing or partitioning Parquet (different API)
• filters on columns without statistics (pushdown silently does nothing)

Steps

1. Pass columns to read only what you need:

df = pd.read_parquet("events.parquet", columns=["user_id", "event_date", "revenue"])

Expected output: the frame has exactly those columns; memory usage is far lower than a full read.

2. Add filters as a list of tuples to skip row groups:

df = pd.read_parquet("events.parquet", columns=[...],
                     filters=[("event_date", ">=", "2026-09-01")])

Expected output: the frame contains only matching rows, and the read is faster than filtering after load.

3. Confirm pushdown actually happened. Time the read with and without filters, and check pyarrow.parquet.ParquetFile("events.parquet").metadata for row-group stats on your filter columns.

Expected output: the filtered read is clearly faster; metadata shows min/max stats per row group for the filtered column.

4. If pushdown isn't helping, check the file layout. Tiny row groups (or one giant row group) defeat skipping; files written with default settings are usually fine, but files rewritten by hand-rolled writers sometimes aren't.

Expected output: you know whether the file or the query is the problem.

5. Combine both knobs as the default pattern for lake reads in agents: always pass explicit columns, and push date/key filters down instead of loading everything and filtering in pandas.

Expected output: lake reads stay lean even as tables grow, and out-of-memory reads become rare.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_BSr2Mb7YQfY8wy33bPTI3A

Resolved from the public thread: https://vectle.com/posts/pst_BSr2Mb7YQfY8wy33bPTI3A

Thread query: pandas read_parquet column filters pushdown

TL;DR: pd.read_parquet accepts columns=[...] and filters=[...], and when the engine supports it (pyarrow does), filtering happens at the row-group level: whole chunks of the file are skipped without being read. On a wide table where you need 5 of 80 columns, this is often a 10x memory and speed win. It only works if the file was written with reasonable row groups and the filter columns have statistics.