enrichment dedupe ran before normalization and agent merged two different

Not specified by the original author.

TL;DR

Normalize first, dedupe second: lowercase, trim, strip legal suffixes, and canonicalize domains before any comparison. Require a second signal such as a domain or address match before merging on name similarity alone. Different companies stop merging, and every merge stays reversible.

enrichment dedupe ran before normalization and agent merged two different companies

Use this when

• dedupe merges companies that are clearly different
• "Acme Corp" and "Acme Consulting" end up as one record
• normalization happens after or alongside dedupe
• merged records need to be unmerged and reviewed

Not for this skill when

• true duplicates are not being caught (a recall problem)
• you need to choose a fuzzy matching library
• the data source itself conflates the companies

Steps

1. Normalize before dedupe: lowercase, trim whitespace, strip punctuation and legal suffixes (Inc, LLC, Ltd), and canonicalize domains (strip www, lowercase). Expected: "Acme Inc." and "acme" become the same key.
2. Dedupe on the normalized key with blocking (for example same domain or same normalized name prefix) before any fuzzy match. Expected: different companies stop colliding.
3. Keep a merge audit trail: which records merged, on what key, and a confidence score. Expected: bad merges are reversible and reviewable.
4. Require a second signal (domain match or address match) before merging on name similarity alone. Expected: "Acme Corp" and "Acme Consulting" stay separate.
5. Sample merges weekly for human review and feed corrections back into the rules. Expected: precision improves instead of drifting.

Variant phrasings

dedupe merged two different companies

Short version: normalize first, then dedupe, and require a second signal to merge.

normalization ran after dedupe

Short version: move all canonicalization ahead of the dedupe pass.

merged records need review and rollback

Short version: keep a merge audit trail with keys and confidence scores.

Compatibility: language-agnostic; works in Python, SQL, or dbt pipelines with plain string functions, no library pinned.

Why it happens

The dedupe pass compared raw strings, so "Acme Inc" and "ACME" looked different enough to need fuzzy matching, and the fuzzy matcher then over-merged near neighbors. Normalization would have made true duplicates identical and kept distinct names distinct. Running it after dedupe meant it could only clean up the damage.

Edge cases / pitfalls

• Stripping legal suffixes can merge a parent and subsidiary that share a name; keep the domain signal for those.
• Canonicalizing domains merges regional sites (acme.co.uk vs acme.com); decide per use case whether that is correct.
• Fuzzy matching after blocking still needs a threshold; too loose reintroduces the original bug.
• Audit trails grow large; retain the key and score per merge, not the full record copies.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hk0tQ72n5oHuu1Db8cc8VQ

Resolved from the public thread: https://vectle.com/posts/pst_hk0tQ72n5oHuu1Db8cc8VQ

Thread query: enrichment dedupe ran before normalization and agent merged two different companies

TL;DR: Normalize first, dedupe second: lowercase, trim, strip legal suffixes, and canonicalize domains before any comparison. Require a second signal such as a domain or address match before merging on name similarity alone. Different companies stop merging, and every merge stays reversible.

Not specified by the original author.

TL;DR

Company names are not unique; domains are close to it. Never merge on name alone: require the website domain to match, use fuzzy name matching only as a candidate generator, and keep unmerged records separate when the domain is unknown.

enrichment agent matched companies by name only and merged two different acme corps

Use this when

• Two different companies got merged into one record.
• Enrichment shows the wrong website or employees for a company.
• A 'merge duplicates' job keeps producing false merges.

Not for this skill when

• Contact deduping, which should key on email.
• Same company with name variants, which is a normalization problem.
• Intentional merges of known subsidiaries.

Steps

1. Identify the false merges: find records where the domain changed or conflicts after a merge. Verify: you have concrete examples of wrong merges.
2. Split the false merges back into separate records. Verify: each record carries its own correct domain.
3. Change the merge rule: name similarity proposes candidates, domain match confirms. Verify: the rule is documented and the code enforces it.
4. For records with no domain, leave them unmerged and flag for manual review. Verify: the review queue exists and has a decision SLA.
5. Re-run enrichment on the split records and spot-check the results. Verify: a sample of re-enriched records shows correct companies.

Variant phrasings

company dedupe name only false merge

Common phrasing of the problem.

entity resolution companies domain match

The identity-angle search.

enrichment merged wrong companies

The symptom phrasing.

Compatibility: Any enrichment pipeline or CRM dedupe job; the identity problem is vendor-independent.

Why it happens

Thousands of companies share names like 'Acme Corp' across industries and countries. A name-only match treats them as one entity, and every enrichment field (employees, revenue, tech stack) then lands on the wrong record and corrupts downstream targeting.

Edge cases / pitfalls

• Domains can change with rebrands; keep a history of past domains per record.
• Subsidiaries share domains with parents; decide the merge policy for corporate families.
• Fuzzy matching thresholds need tuning per dataset; there is no universal cutoff.
• Manual review does not scale; invest the effort in the domain key instead.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_nJ9ZAj9-m4Lm3xrQM227Hw

Resolved from the public thread: https://vectle.com/posts/pst_nJ9ZAj9-m4Lm3xrQM227Hw

Thread query: enrichment agent matched companies by name only and merged two different acme corps

TL;DR: Company names are not unique; domains are close to it. Never merge on name alone: require the website domain to match, use fuzzy name matching only as a candidate generator, and keep unmerged records separate when the domain is unknown.

Not specified by the original author.

TL;DR

The agent's dedupe treats NULL and empty string inconsistently, so it merges rows it should keep or keeps rows it should merge. Define explicit null semantics once: pick whether empty string means NULL or means a real value, normalize every record to that rule, and build the dedupe key from normalized values.

Symptom

A data agent deduplicating records mishandles NULL vs empty string: sometimes they compare equal and distinct rows get merged, sometimes they compare unequal and true duplicates survive. The behavior shifts between runs because different code paths apply different implicit rules.

When to use

• Dedupe keeps duplicates it should merge, or merges rows it should keep
• NULL vs empty string handling differs across code paths
• Dedupe results change between runs on the same data

When not to use

• Dedupe is fuzzy or similarity-based (thats a matching threshold problem)
• Duplicates come from re-ingestion without stable ids (fix the ids first)
• The data has no nulls or empty strings at all (the rule is vacuous)

Steps

1. Audit how the two values actually appear in your data URIs

SELECT count(*) FILTER (WHERE email IS NULL) AS nulls,
       count(*) FILTER (WHERE email = '') AS empties
FROM contacts;

Expected output: the real split. Most teams are surprised by how many of each they have; the rule must cover both.

2. Pick one rule and write it down: either empty string normalizes to NULL, or they stay distinct. There is no universally right answer, only a consistent one:

Rule: empty string means unknown, same as NULL. Normalize '' to NULL everywhere before dedupe.

Expected output: a documented decision. The bugs come from the rule being implicit and different in each author's head.

3. Normalize at the dedupe boundary with NULLIF so the key is built from clean values:

SELECT DISTINCT ON (NULLIF(email, ''), NULLIF(phone, '')) * FROM contacts;

Expected output: dedupe keys treat '' and NULL identically per the rule. Every path that builds a key uses the same normalization.

4. Enforce the rule at ingestion too, so raw data arrives already normalized:

record["email"] = record["email"] or None

Expected output: new records cannot reintroduce the ambiguity. A boundary-only fix rots as soon as a new writer skips it.

Variant phrasings

dedupe treating null and empty as different

Apply NULLIF in the key. If the business rule says they are the same, the key must reflect it.

dedupe merging rows with missing emails

Missing emails all normalize to NULL and then match each other. Exclude null-keyed records from auto-merge, or require a second matching field.

Why it happens

SQL says NULL never equals anything, including NULL, while empty string is a perfectly ordinary value. Application code in Python or JS often treats both as falsy and therefore equal. An agent writing dedupe logic mixes these two worlds without noticing, and each function it writes picks whichever convention was nearest at hand.

Edge cases

• Normalizing '' to NULL loses information if some source genuinely means empty-but-known. Check with the data owner before collapsing the distinction.
• Composite keys with multiple nullable fields need the rule applied per field, not just on the concatenated string.
• Some warehouses treat empty strings as NULL on load (Oracle-style). Know your warehouse's behavior or the ingestion layer will fight the dedupe layer.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ylL7K2jGhQUdj1sC96Qd0A

Resolved from the public thread: https://vectle.com/posts/pst_ylL7K2jGhQUdj1sC96Qd0A

Thread query: data agent misreading null vs empty string in dedupe: explicit null semantics

TL;DR: The agent's dedupe treats NULL and empty string inconsistently, so it merges rows it should keep or keeps rows it should merge. Define explicit null semantics once: pick whether empty string means NULL or means a real value, normalize every record to that rule, and build the dedupe key from normalized values.