agent timed out on the transaction-match loop: 120k lines and the fuzzy match never converged

TL;DR

Fuzzy matching everything against everything on 120k lines is a computation that cannot finish: the pair space is in the billions. Stage the match: exact keys first, which clears most lines cheaply, then deterministic rules, and reserve fuzzy scoring for the residue, blocked into candidate groups by amount, date window, or counterparty so comparisons happen only where a match is plausible. Add a convergence budget per stage and persist results per stage, so the loop's progress is real and bounded.

agent timed out on the transaction-match loop: 120k lines and the fuzzy match never converged

Use this when

• The match job times out or runs indefinitely on about 120k lines.
• Fuzzy or similarity scoring is applied broadly rather than to a reduced candidate set.
• Exact-matchable pairs are still unmatched when the job dies, showing staging is absent.

Not for this skill when

• The match set is small but one pathological pair loops the scorer. That is a scorer bug, not a scale design issue.
• The job is slow because it calls an external service per pair. Batch or cache that call; the staging advice still helps but the bottleneck is elsewhere.

Steps

1. Measure the current state: how many lines exact keys alone can match, tested on a sample with amounts, dates, and references. Verify: the exact-match yield is quantified before redesigning
2. Rebuild the pipeline in stages: exact, then rule-based, then fuzzy on the residue only, persisting results after each stage. Verify: each stage's input shrinks and its output is durable
3. Block fuzzy candidates by amount equality or tight tolerance plus a date window, indexing lines so candidate retrieval is not a full scan. Verify: candidate pairs per line drop to a small average number
4. Set per-stage budgets: maximum pairs scored, maximum runtime, and a stop that reports residue rather than looping. Verify: the fuzzy stage terminates by budget with a stated residue
5. Run the full 120k set through the staged pipeline and report match rates per stage and total runtime. Verify: the run completes and the fuzzy residue is a small, reviewable exception list

Variant phrasings

fuzzy match 120k transactions never converges timeout

The non-converging match phrasing.

transaction matching staged exact rule fuzzy blocking

When staging the pipeline.

Compatibility: Agent matching engines over ERP and bank transaction sets at six-figure line counts.

Why it happens

Matching feels like one problem but is two: finding candidate pairs, and judging them. Naive fuzzy loops fuse the two and pay judging costs for pairs no human would consider, such as different amounts in different months. Blocking by the attributes every true match shares collapses the candidate space by orders of magnitude, and staging spends expensive judgement only on lines cheap methods could not settle. Convergence is then a property of the design, not a hope.

Edge cases / pitfalls

• Amount-plus-date blocking misses matches where fees change the amount; add a second blocking pass on reference tokens for the residue.
• Persisting per stage also creates the audit story: this line matched exactly, that one needed fuzzy judgement at a stated score.
• If volumes keep growing, the fuzzy stage itself can be sampled for quality rather than expanded; precision on exceptions matters more than squeezing the last percent.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_myBOfipEgzkopW3tD1Fmxg