how to profile pandas memory usage per column

TL;DR

Run df.memory_usage(deep=True) to see bytes per column, sort it, and attack the top offenders: downcast numerics, convert repeated strings to category, and drop what you dont need. deep=True is the key detail; without it, object columns report only pointer sizes and lie about their real cost.

df.memory_usage(deep=True).sort_values(ascending=False)

Use this when

• A DataFrame uses more RAM than expected
• You need to find which columns to downcast or drop
• Before/after comparison for a dtype optimization pass

Not for

• MemoryError on files that dont fit in RAM, use the chunking skill
• Profiling Python code outside pandas
• CPU/runtime profiling, use cProfile or line_profiler

Steps

1. Get per-column usage with deep introspection:

usage = df.memory_usage(deep=True, index=True).sort_values(ascending=False)
print(usage)

Expected output: bytes per column, index included, biggest first.

1. Compare against the shallow number to see hidden string costs:

df.memory_usage(deep=False).sum(), df.memory_usage(deep=True).sum()

Expected output: two totals. A big gap means object columns hold most of the real memory.

1. Downcast the numeric offenders:

df['n'] = pd.to_numeric(df['n'], downcast='integer')
df['f'] = pd.to_numeric(df['f'], downcast='float')

Expected output: int64 to int32/int16/int8 where values fit; rerun step 1 to confirm the drop.

1. Convert repeated-string columns to category:

df['region'] = df['region'].astype('category')

Expected output: the column shrinks to roughly one small int per row plus the category table.

1. Report the total win:

print(f"{(before - after) / before:.0%} smaller")

Expected output: a percentage. 50-80 percent is typical on messy real-world frames.

Variant phrasings

pandas dataframe memory usage by column

df.memory_usage(deep=True) is the one-liner. Sort it and work top-down.

why is my dataframe so large pandas

Usually object-dtype strings and int64/float64 defaults. Steps 3-4 fix both.

reduce pandas dataframe memory footprint

Profile first (step 1), then downcast, categorize, and drop unused columns, in that order.

Why it happens

Pandas defaults are generous: int64 for any integer, float64 for any float, and full Python objects for strings. Each is 2-8x bigger than the data needs. deep=False hides the string cost because it only counts the 8-byte pointers, which is why people underestimate object columns.

Edge cases

• deep=True is slower on huge frames; sample 100k rows for a quick estimate.
• Category dtype has overhead per category; dont use it for near-unique strings like IDs or free text.
• Downcasting can overflow: verify df['n'].max() fits the target dtype before committing.
• The index itself can be big (a large string index); set_index on an int column or reset_index to a RangeIndex helps.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_daeY-YYVANuMjuRBZU-CgA