agent's hubspot search failed when agent tried fuzzy matching 100k records
Fixes an agent's HubSpot search failing on 100k-record fuzzy matching: push matching server-side with search filters, batch the candidates, and never pull all records to fuzzy-match locally. Use when fuzzy match jobs fail. Not for exact-match searches.
TL;DR
Pulling 100k records to fuzzy-match locally is slow, memory-hungry, and rate-limit bait. Instead, use HubSpot search with tight filters to pull small candidate sets per record, then fuzzy-match only within each set. The search API does the coarse filtering; your code does the fine matching.
Error
Agent failed: search requests throttled and the local match job ran out of memory at 62,000 records.Steps
- For each source record, build a HubSpot search with strong filters (email domain, phone digits, name token). Expected: tens of candidates per record, not thousands.
- Page the candidates (they will be few) and fuzzy-match locally within that set. Expected: fast, low-memory matching.
- Cache candidate sets for repeated runs. Expected: reruns skip the search phase.
- Pace the searches to stay under the search rate limits. Expected: no 429s.
- Log match scores and the winning candidate for audit. Expected: every merge decision is explainable.
When to use
- Fuzzy matching jobs fail on large record sets.
- An agent downloads everything to match locally.
- Search throttling kills the match run.
When not to use
- Exact email or id matching (no fuzzy logic needed).
- Small sets where local matching is fine.
Tool compatibility
- HubSpot CRM search API v3; local fuzzy matching libraries.
- Blocking keys to narrow candidates.
Variant phrasings
search throttled during matching
Narrow candidates per record and pace.
out of memory fuzzy matching
Do the coarse filter server-side.
Why it happens
Fuzzy matching is O(n*m) without blocking. At 100k records that is billions of comparisons, plus the API cost of fetching everything. Server-side search cuts the problem by orders of magnitude.
Edge cases
- Overly tight filters miss true matches; validate recall on a labeled sample.
- Search indexes lag real-time writes by a bit; very fresh records may not appear.
- Score thresholds need tuning per field; names need looser thresholds than emails.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstwgSaCGUG6Ewl73kWMukJw