enrichment dedupe ran before normalization and agent merged two different companies

TL;DR

Normalize first, dedupe second: lowercase, trim, strip legal suffixes, and canonicalize domains before any comparison. Require a second signal such as a domain or address match before merging on name similarity alone. Different companies stop merging, and every merge stays reversible.

enrichment dedupe ran before normalization and agent merged two different companies

Use this when

• dedupe merges companies that are clearly different
• "Acme Corp" and "Acme Consulting" end up as one record
• normalization happens after or alongside dedupe
• merged records need to be unmerged and reviewed

Not for this skill when

• true duplicates are not being caught (a recall problem)
• you need to choose a fuzzy matching library
• the data source itself conflates the companies

Steps

1. Normalize before dedupe: lowercase, trim whitespace, strip punctuation and legal suffixes (Inc, LLC, Ltd), and canonicalize domains (strip www, lowercase). Expected: "Acme Inc." and "acme" become the same key.
2. Dedupe on the normalized key with blocking (for example same domain or same normalized name prefix) before any fuzzy match. Expected: different companies stop colliding.
3. Keep a merge audit trail: which records merged, on what key, and a confidence score. Expected: bad merges are reversible and reviewable.
4. Require a second signal (domain match or address match) before merging on name similarity alone. Expected: "Acme Corp" and "Acme Consulting" stay separate.
5. Sample merges weekly for human review and feed corrections back into the rules. Expected: precision improves instead of drifting.

Variant phrasings

dedupe merged two different companies

Short version: normalize first, then dedupe, and require a second signal to merge.

normalization ran after dedupe

Short version: move all canonicalization ahead of the dedupe pass.

merged records need review and rollback

Short version: keep a merge audit trail with keys and confidence scores.

Compatibility: language-agnostic; works in Python, SQL, or dbt pipelines with plain string functions, no library pinned.

Why it happens

The dedupe pass compared raw strings, so "Acme Inc" and "ACME" looked different enough to need fuzzy matching, and the fuzzy matcher then over-merged near neighbors. Normalization would have made true duplicates identical and kept distinct names distinct. Running it after dedupe meant it could only clean up the damage.

Edge cases / pitfalls

• Stripping legal suffixes can merge a parent and subsidiary that share a name; keep the domain signal for those.
• Canonicalizing domains merges regional sites (acme.co.uk vs acme.com); decide per use case whether that is correct.
• Fuzzy matching after blocking still needs a threshold; too loose reintroduces the original bug.
• Audit trails grow large; retain the key and score per merge, not the full record copies.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hk0tQ72n5oHuu1Db8cc8VQ