Splink deduplication failed error "blocking rule generated too many comparisons"

TL;DR

Loosen the blocking before tuning the matching: Splink blocking rules that generate too many comparisons are too broad for the data size, so add a more selective blocking key, or combine keys, until the comparison count is tractable, then run the model. Blocking decides which record pairs are compared at all. A rule on a common field, such as a city or a year, pairs millions of unrelated records and the job dies before matching starts.

Splink deduplication failed error "blocking rule generated too many comparisons"

Use this when

• The check or run fails with this symptom: a Splink deduplication fails with blocking rule generated too many comparisons.
• You are working on a Splink linkage or deduplication job over a large table of people or organisations and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A small dataset where too many comparisons signals skewed values, such as a placeholder name shared by thousands of records.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Read the blocking rules and the estimated comparison count per rule. Verify: the rule producing the bulk of comparisons is identified
2. Check the value distribution of that rule fields. Verify: one very common value explains the blow-up, or the key is uniformly weak
3. Tighten the rule by adding a second field, such as a name token with a birth year, or drop the weakest rule. Verify: the estimated count falls to a workable size
4. Handle placeholder values explicitly by excluding them from blocking. Verify: placeholder driven pairs disappear from the estimate
5. Run the Splink job. Verify: comparisons complete and clustering produces results

Variant phrasings

splink blocking rule generated too many comparisons

The shorter phrasing focused on the named failure.

splink deduplication failed blocking comparisons

The symptom phrasing a searcher tries next.

splink blocking too many comparisons fix

The variant naming the failing stage or tool directly.

Compatibility: Splink 3.x and 4.x blocking rules and comparison estimates

Why it happens

Blocking trades recall for tractability. Rules written on a sample, where a city block held dozens of records, explode on production data where the same block holds hundreds of thousands, because pair counts grow with the square of the block size.

Edge cases / pitfalls

• Tightening blocking loses some true matches at the edges; measure the recall effect on a labelled sample before locking the rules.
• Salting large blocks is an alternative when a broad rule carries unique recall value.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XTsqaHLLskk-SZYinC2fzg

Resolved from the public thread: https://vectle.com/posts/pst_XTsqaHLLskk-SZYinC2fzg

Thread query: Splink deduplication failed error "blocking rule generated too many comparisons"

TL;DR: Loosen the blocking before tuning the matching: Splink blocking rules that generate too many comparisons are too broad for the data size, so add a more selective blocking key, or combine keys, until the comparison count is tractable, then run the model. Blocking decides which record pairs are compared at all. A rule on a common field, such as a city or a year, pairs millions of unrelated records and the job dies before matching starts.