agent ran out of memory and crashed building the pairwise match matrix for a 90k-line month

TL;DR

Ninety thousand lines squared is eight billion pairs; no machine holds that matrix. Never build it. Generate candidates only within blocks, lines sharing amount and a date window, score and decide each block, then discard the block and write decisions out. Keep only line-level data and compact indexes in memory, and persist results incrementally. The match quality comes from blocking design, and the memory footprint becomes proportional to one block, not to the month.

agent ran out of memory and crashed building the pairwise match matrix for a 90k-line month

Use this when

• The job dies on memory while building or holding candidate pairs for about 90k lines.
• Memory growth is proportional to lines squared, not to line count.
• Blocking or indexing is absent; every line is compared with every other line.

Not for this skill when

• Memory grows with line count linearly and still exhausts; suspect a leak or retained objects per line, not the pair matrix.
• The job loads 90k full transaction objects with histories. Trimming object size may help but the matrix logic still needs blocking.

Steps

1. Measure the pair count the matrix implies for 90k lines and the per-pair memory cost. Verify: the arithmetic shows the matrix cannot fit, confirming the diagnosis
2. Design blocking keys: exact or near amount plus a date window, plus a reference token pass for the residue. Verify: each true match in a test sample shares a block with its partner
3. Rewrite candidate generation to emit one block at a time, score pairs inside the block, write decisions, and release the block. Verify: peak memory stays flat as more blocks are processed
4. Persist match decisions incrementally so an interruption loses at most the current block. Verify: killing a test run leaves all completed blocks decided
5. Run the full 90k month and report peak memory, blocks processed, match rate, and residue. Verify: the run completes within the platform memory budget with measured headroom

Variant phrasings

pairwise match matrix out of memory 90k lines blocking

The matrix-OOM phrasing.

agent matching memory candidate generation blocks streaming

When rebuilding candidate generation.

Compatibility: Agent matching pipelines at high line counts on memory-constrained runners.

Why it happens

Memory is the silent dimension of matching algorithms. Developers test on thousands of lines where a dense matrix fits, and production delivers tens of thousands where it does not, a failure that arrives only at month end. Blocking is the standard answer because reconciliation truth lives in shared attributes: a real match almost always agrees on amount and rough date. The matrix is not needed; the blocks are where the answers are.

Edge cases / pitfalls

• Calibrate block width with samples: too narrow loses real matches to fees and date drift, too wide recreates the memory problem.
• An index over block keys avoids scanning all lines to build each block; use dictionaries keyed by the blocking attributes.
• Spill blocks to disk if a single block still spikes memory for pathological amounts like 0.00 or 1.00 shared by many lines.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_udYY8JW3svjh6JyRCPLDCQ