polars ShapeError: unable to hstack dataframes

TL;DR

hstack glues dataframes side by side and requires identical row counts; yours differ because one side was filtered, sampled, or came from a different source. Align the row counts first (filter both the same way, or join on a key instead of stacking), then hstack.

polars ShapeError: unable to hstack dataframes

Use this when

• df1.hstack(df2) or horizontal concat raises ShapeError
• An agent builds features in two frames and combines them
• The error appeared after adding a filter to one side

Not for this skill when

• Vertical concat fails (thats a schema mismatch)
• A join returns wrong rows (thats the join key)
• Columns clash by name (thats a rename problem)

Steps

1. Print both shapes to see the gap:

print(df1.shape, df2.shape)

Expected output: something like (1000, 5) (997, 3). The 3-row gap is what hstack refuses to paper over.

2. Find where the rows went. Usually one side was filtered:

# check: was df2 filtered after df1 was built?
print(df2_raw.shape)  # the pre-filter shape, if you kept it

Expected output: you identify the filter that dropped the 3 rows. Apply the same filter to both sides, or filter before splitting into two frames.

3. If the frames genuinely describe the same rows in the same order, align and stack:

# option A: filter both identically, then
combined = df1.hstack(df2)

Expected output: the stack succeeds. This is only valid when row order is meaningful and identical on both sides.

4. If row identity matters, join on a key instead of stacking:

combined = df1.join(df2, on="id", how="inner")

Expected output: rows matched by key, no shape requirement. hstack is positional; join is logical. When in doubt, join.

Variant phrasings

horizontal concat fails after drop_nulls on one frame

Thats the filter case (step 2). drop_nulls changed one side's row count; apply it to both or join.

shapes match but hstack still fails

Check for duplicate column names; hstack also rejects those. Rename before stacking.

Why it happens

hstack is a positional operation: row i of the left meets row i of the right. Polars refuses to guess what to do with the extra rows, so mismatched heights are a hard error rather than silent null-padding. Agent pipelines hit it because feature engineering steps filter or sample one frame while the label frame keeps its original rows.

Edge cases

• After any shuffle, positional alignment is meaningless; join on an id column instead.
• Lazy frames: shapes arent known until collect, so the error surfaces late. Collect shapes early when debugging.
• If you must pad, do it explicitly with a null frame of the right height so the padding is visible in code review.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_zvmBe6XdSp77auK0DYOF8g

Resolved from the public thread: https://vectle.com/posts/pst_zvmBe6XdSp77auK0DYOF8g

Thread query: polars ShapeError: unable to hstack dataframes

TL;DR: hstack glues dataframes side by side and requires identical row counts; yours differ because one side was filtered, sampled, or came from a different source. Align the row counts first (filter both the same way, or join on a key instead of stacking), then hstack.