polars string cache for categorical joins

TL;DR

In Polars, Categorical columns use integer codes that are only meaningful within a shared string cache. If two frames built their categoricals under different caches (or none), joining on them errors out or, worse, silently mismatches. The fix: call pl.enable_string_cache() once at the start of your program, before any categorical is created, so every categorical shares one global mapping. Alternatively, cast both join keys to the same categorical explicitly.

The query

polars string cache for categorical joins

Use this when

• a join on categorical columns raises a StringCache mismatch error
• categoricals come from different files or different parts of a pipeline
• a data agent builds Polars frames and joins them across function boundaries

Not for

• joins on plain String columns (no cache involved)
• pandas categoricals (different machinery)
• enum types (Polars Enum has its own fixed ordering)

Steps

1. Enable the global string cache at program start, before creating any categorical column:

pl.enable_string_cache()

Expected output: subsequent .cast(pl.Categorical) calls share one global mapping.

2. Alternatively, use it as a context manager around the section that builds and joins categoricals:

with pl.StringCache():
    # build frames and join here

Expected output: the cache is active exactly where needed and cleaned up after.

3. If you can't control creation order, cast both sides to categorical inside the cache right before the join. Both frames must map through the same cache for the codes to agree.

Expected output: the join succeeds and matches on actual string values, not stale codes.

4. Verify the join result, not just its success. A mismatched-cache join can silently produce wrong matches in some versions; check a few known keys by hand.

Expected output: spot-checked keys match the same rows they would under a String join.

5. Consider whether you need Categorical at all. If the column has high cardinality or the join is one-off, plain String joins skip the whole issue. Categorical pays off for repeated low-cardinality group-bys and joins.

Expected output: a deliberate choice per column instead of categorical-by-default.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RHfS7fVVPQ8BeB3WTq2hGg

Resolved from the public thread: https://vectle.com/posts/pst_RHfS7fVVPQ8BeB3WTq2hGg

Thread query: polars string cache for categorical joins

TL;DR: In Polars, Categorical columns use integer codes that are only meaningful within a shared string cache. If two frames built their categoricals under different caches (or none), joining on them errors out or, worse, silently mismatches. The fix: call pl.enable_string_cache() once at the start of your program, before any categorical is created, so every categorical shares one global mapping. Alternatively, cast both join keys to the same categorical explicitly.