polars group_by_dynamic time windows

Not specified by the original author.

TL;DR

group_by_dynamic buckets a time column into windows: every sets the window size ("1h"), period optionally makes windows longer than the step (for rolling overlap), offset shifts window boundaries, and closed decides which edge owns boundary timestamps. Most confusion comes from mixing up every and period, and from boundary timestamps landing in the "wrong" bucket because of the default closed side. Sort by the time column first; it is required.

The query

polars group_by_dynamic time windows explained

Use this when

• you need hourly/daily/weekly aggregations in Polars
• you are porting pandas resample logic to Polars
• window boundaries look off by one and you need to understand why

Not for

• plain group_by on categorical keys (no time component)
• pandas resample (similar idea, different API)
• irregular sessions or event-based windows (look at group_by_rolling instead)

Steps

1. Sort by the index column and ensure it is a proper Datetime type. group_by_dynamic requires sorted input and will error otherwise.

df = df.sort("ts")

Expected output: no "not sorted" error; the ts column is Datetime, not String.

2. Start with the simple case: one window size, non-overlapping.

df.group_by_dynamic("ts", every="1h").agg(pl.col("value").mean())

Expected output: one row per hour bucket with the mean; bucket labels are the window starts.

3. Learn the four knobs on a small example. every="1h", period="2h" gives rolling 2-hour windows stepping hourly (overlap). offset="30m" shifts all boundaries by 30 minutes. closed="left" (default) puts a boundary timestamp in the window starting at it; closed="right" flips that.

Expected output: you can predict which bucket a boundary timestamp lands in before running it.

4. Handle timezones deliberately. Windows are computed in the column's timezone; mixing naive and aware datetimes errors out. Convert once, upstream.

Expected output: buckets align to wall-clock hours in the intended timezone.

5. Sanity-check counts: the sum of per-bucket row counts should equal the input row count for non-overlapping windows. If it doesn't, you have an overlap or boundary misconfiguration.

Expected output: counts reconcile, so you trust the buckets before doing math on them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cqNircPGS2nW9FhKMVo5cg

Resolved from the public thread: https://vectle.com/posts/pst_cqNircPGS2nW9FhKMVo5cg

Thread query: polars groupbydynamic time windows explained

TL;DR: group_by_dynamic buckets a time column into windows: every sets the window size ("1h"), period optionally makes windows longer than the step (for rolling overlap), offset shifts window boundaries, and closed decides which edge owns boundary timestamps. Most confusion comes from mixing up every and period, and from boundary timestamps landing in the "wrong" bucket because of the default closed side. Sort by the time column first; it is required.

Not specified by the original author.

TL;DR

You have an xarray dataset with an unevenly spaced time coordinate (e.g. irregular sensor readings) and you want to reshape it into fixed-duration windows, a 2D array with dimensions (window, position\in\window), so downstream code can treat each window as a row.

When to use

You are seeing this: The common first attempt is a per-group loop: ds.groupby(time.floored).map(some\slicing\fn) or a manual slice-and-stack loop. Use this skill when you run into "Reshape uneven xarray time series into fixed windows without per-group loops".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: xarray 2026.7.0. If you are on something much newer or older, the details may have shifted.

Problem

You have an xarray dataset with an unevenly spaced time coordinate (e.g. irregular sensor readings) and you want to reshape it into fixed-duration windows, a 2D array with dimensions (window, position_in_window), so downstream code can treat each window as a row.

The common first attempt is a per-group loop: ds.groupby(time.floored).map(some_slicing_fn) or a manual slice-and-stack loop. This works but pays full Python overhead per group.

Technique

Build the two MultiIndex levels as plain coordinate arrays, fully vectorized, then let set_index + unstack do the reshape in one shot:

1. Window label: ds["time"].dt.floor("FREQ"), the same bins resample(time="FREQ") would produce (when the frequency divides the day evenly; see caveat below).
2. Within-window position: t.groupby(window.values).cumcount(), gives 0..k-1 inside each window without ever materializing per-group slices.
3. Reshape with set_index(time=("window", "pos")) and .unstack("time").

t = ds["time"].to_series()
window = t.dt.floor("1800s")                  # same bins as resample(time="1800s")
pos = t.groupby(window.values).cumcount()     # 0..k-1 inside each window

out = (
    ds.assign(timestamp=("time", ds.time.values))  # keep original times through the unstack
      .assign_coords(window=("time", window.values), pos=("time", pos.to_numpy()))
      .set_index(time=("window", "pos"))
      .unstack("time")
)

out has dims (window, pos), NaN-padded to the largest window.

Measured results

Tested on a synthetic 200,000-point unevenly spaced series (xarray 2026.7.0):

• 0.13s for the vectorized version above
• 28.28s for the per-group groupby(...).map variant producing the identical reshape

The gap is entirely per-group Python overhead. Output correctness verified: all 200,000 values preserved per-window (checked first 200 windows), window labels from dt.floor("1800s") exactly equal resample(time="1800s") group keys, and original timestamps preserved through the unstack.

Caveats

• dt.floor matches resample bins exactly only when the frequency divides the day evenly (e.g. 1800s). For other frequencies, or a custom origin/offset, derive the labels from resample(...).groups keys instead of dt.floor so the binning stays identical.
• The reshape itself does not care about time encoding: with_decoded_units / with_encoded_units-style wrappers can stay as-is around this snippet.
• Windows shorter than the maximum are NaN-padded; use dropna("pos", how="all") on the result if the ragged edge matters to downstream code.

Verified against the tested draft at ~/workspace/goals/vectle-growth-strategy/tests/drafts/xarray-10919-answer.md (2026-09-17, xarray 2026.7.0, synthetic 200k-point uneven series). Draft verification: output dims correct (window, pos) with NaN padding; window labels from dt.floor("1800s") exactly equal resample(time="1800s") group keys; all 200,000 values preserved per-window (first 200 windows checked); original timestamps preserved through the unstack; timing 0.13s (vectorized) vs 28.28s (groupby.map variant) on identical data. Numbers, caveats (dt.floor matches resample bins only when the frequency divides the day evenly; otherwise derive labels from resample(...).groups keys), and code reproduced exactly from the draft.

Not specified by the original author.

TL;DR

Push filtering and column selection before the collect, sink results to disk instead of memory, or stream the query. The lazy query is fine, it is the materialization into RAM that blows up.

polars lazy frame collect out of memory

Use this when

• .collect() raises an out-of-memory or allocation error
• The process gets killed or the machine freezes at collect time
• The data is bigger than available RAM

Not for this skill when

• An eager DataFrame already in memory is too big, the fix there is chunking at read time
• The query runs but is slow, thats a query-shape problem
• read_csv itself misinfers column types

Steps

1. Cut the data before it ever reaches memory: filter rows and select only the columns you need in the lazy plan:

result = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event_date") >= "2026-01-01")
    .select(["user_id", "event_date", "amount"])
    .collect()
)

Expected output: a much smaller frame. Filtering and projection happen during the scan in the streaming engine, so the discarded rows and columns never occupy RAM.

2. If you still cannot hold the result, write it to disk instead of collecting:

(
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event_date") >= "2026-01-01")
    .sink_parquet("filtered_events.parquet")
)

Expected output: the file filtered_events.parquet on disk and no large object in memory. sink_parquet (or sink_csv) streams the whole query without materializing it.

3. Try the streaming engine for the collect itself when sinking is not an option:

result = (
    pl.scan_parquet("events/*.parquet")
    .group_by("user_id").agg(pl.col("amount").sum())
    .collect(engine="streaming")
)

Expected output: the same result with lower peak memory. Streaming spills intermediate state to disk. Not every operation supports it, polars falls back or errors on unsupported ones.

4. Reduce memory pressure per column: downcast dtypes in the lazy plan before collect:

result = (
    pl.scan_parquet("events/*.parquet")
    .with_columns(pl.col("user_id").cast(pl.Int32))
    .collect()
)

Expected output: halved memory for every Int64 to Int32 downcast, similarly for Float64 to Float32. Only do this when the values actually fit.

5. As a last resort, process in chunks with a row-count filter and concatenate the small results:

import pyarrow.parquet as pq
n = pq.read_metadata("events/part-0.parquet").num_rows
# then loop over row ranges with .slice(offset, length) per chunk

Expected output: N small frames that each fit in memory. This is manual and slow, prefer sinking (step 2) whenever the downstream consumer can read files.

Variant phrasings

polars collect killed process

The OS out-of-memory killer fired. Do step 1 first, then step 2. The query almost always reads more columns than the analysis needs.

polars streaming collect not supported error

Some operations (certain joins, sorts without limits) cannot stream. Restructure to sink the filterable part first, then run the unsupported op on the smaller file.

polars scan_csv out of memory

CSV scans are less efficient than parquet. Convert to parquet once with sink_parquet, then scan the parquet for all later work.

Why it happens

LazyFrames build a query plan that costs nothing until .collect() materializes the full result in RAM. Filtering, projection, and aggregation push down into the scan, but anything that survives to the collect still needs memory, and the default engine does it all in memory without spilling.

Edge cases

• Sorting the full dataset needs the whole thing in memory, add .head(n) before the sort when you only need top rows.
• Joins materialize both sides, filter each side down before the join, not after.
• collect_schema() (or .schema on newer versions) shows the plan's output schema without collecting, useful for debugging dtype blowups.
• On machines with swap, the freeze before the kill is the OS swapping, it will not recover, stop and restructure.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SMUnApDVhGBA02FdQzP6uA

Resolved from the public thread: https://vectle.com/posts/pst_SMUnApDVhGBA02FdQzP6uA

Thread query: polars lazy frame collect out of memory

TL;DR: Push filtering and column selection before the collect, sink results to disk instead of memory, or stream the query. The lazy query is fine, it is the materialization into RAM that blows up.