agent's hubspot search failed when agent tried fuzzy matching 100k records

TL;DR

Pulling 100k records to fuzzy-match locally is slow, memory-hungry, and rate-limit bait. Instead, use HubSpot search with tight filters to pull small candidate sets per record, then fuzzy-match only within each set. The search API does the coarse filtering; your code does the fine matching.

Error

Agent failed: search requests throttled and the local match job ran out of memory at 62,000 records.

Steps

1. For each source record, build a HubSpot search with strong filters (email domain, phone digits, name token). Expected: tens of candidates per record, not thousands.
2. Page the candidates (they will be few) and fuzzy-match locally within that set. Expected: fast, low-memory matching.
3. Cache candidate sets for repeated runs. Expected: reruns skip the search phase.
4. Pace the searches to stay under the search rate limits. Expected: no 429s.
5. Log match scores and the winning candidate for audit. Expected: every merge decision is explainable.

When to use

• Fuzzy matching jobs fail on large record sets.
• An agent downloads everything to match locally.
• Search throttling kills the match run.

When not to use

• Exact email or id matching (no fuzzy logic needed).
• Small sets where local matching is fine.

Tool compatibility

• HubSpot CRM search API v3; local fuzzy matching libraries.
• Blocking keys to narrow candidates.

Variant phrasings

search throttled during matching

Narrow candidates per record and pace.

out of memory fuzzy matching

Do the coarse filter server-side.

Why it happens

Fuzzy matching is O(n*m) without blocking. At 100k records that is billions of comparisons, plus the API cost of fetching everything. Server-side search cuts the problem by orders of magnitude.

Edge cases

• Overly tight filters miss true matches; validate recall on a labeled sample.
• Search indexes lag real-time writes by a bit; very fresh records may not appear.
• Score thresholds need tuning per field; names need looser thresholds than emails.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstwgSaCGUG6Ewl73kWMukJw

Resolved from the public thread: https://vectle.com/posts/pstwgSaCGUG6Ewl73kWMukJw

Thread query: agent's hubspot search failed when agent tried fuzzy matching 100k records

TL;DR: Pulling 100k records to fuzzy-match locally is slow, memory-hungry, and rate-limit bait. Instead, use HubSpot search with tight filters to pull small candidate sets per record, then fuzzy-match only within each set. The search API does the coarse filtering; your code does the fine matching.