how to fix pandas "cannot reindex on an axis with duplicate labels"

TL;DR

This error means your index has repeated labels and pandas refuses to reindex or align along it. Fix it by deduplicating the index first, usually with df = df[~df.index.duplicated(keep='first')], then rerun your reindex. It happens because reindex needs a one-to-one mapping between labels, and duplicates make that ambiguous.

ValueError: cannot reindex on an axis with duplicate labels

Use this when

• df.reindex(...) raises ValueError: cannot reindex on an axis with duplicate labels
• .loc assignment across a duplicated index errors the same way
• You stacked, concatenated, or merged frames and the index now repeats

Not for

• Duplicate rows (not index labels), use a dedupe-rows approach
• pivot_table duplicate-entries errors, different fix
• Merge key mismatches, use the merge skill

Steps

1. Confirm the index is duplicated:

df.index.duplicated().sum()

Expected output: a number greater than 0. If it is 0, the problem is elsewhere.

1. Look at the repeated labels:

df.index[df.index.duplicated(keep=False)].unique()

Expected output: the label values that repeat.

1. Drop the duplicates (keep first occurrence):

df = df[~df.index.duplicated(keep='first')]

Expected output: df.index.duplicated().sum() now returns 0.

1. Rerun your original reindex:

df.reindex(new_index)

Expected output: a reindexed frame with no error, NaN where new labels had no match.

1. If you need to KEEP duplicates (e.g. time-series with repeated timestamps), reset to a unique integer index instead:

df = df.reset_index(drop=False)

Expected output: the old index becomes a regular column, new RangeIndex has no duplicates.

Variant phrasings

pandas reindex duplicate index error

Same thing. The axis pandas is complaining about is usually the row index, sometimes the columns (df.columns.duplicated()).

ValueError cannot reindex from a duplicate axis

The newer pandas wording. Identical fix: dedupe first or reset the index.

reindex fails after concat or merge

pd.concat of frames with overlapping indexes is the most common source. Either pass ignore_index=True to concat, or dedupe after.

Why it happens

Reindexing maps every label in the new index to exactly one position in the old one. A repeated label breaks that mapping: pandas would have to guess which row you mean, so it refuses. Concat, merge, and append are the usual ways duplicates sneak in.

Edge cases

• Duplicated COLUMNS (not the row index): check df.columns.duplicated().any() and rename or drop via df.loc[:, ~df.columns.duplicated()].
• If duplicates are legitimate (repeated timestamps), dont silently drop them, use reset_index and keep the values in a column.
• verify_integrity=True on pd.concat catches this at concat time instead of at reindex time.
• MultiIndex duplicates: use df.index.duplicated() the same way, it works on MultiIndex too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uqE5TU1V2Lz4MgrdV79taQ