locale qa agent crashed on xliff file over 50mb and the run failed

Fix the locale QA agent crashing on an XLIFF file over 50MB

TL;DR

The agent loads the whole XLIFF into memory, so stream it instead. A 50MB XML file becomes hundreds of megabytes as a DOM, and the agent OOMs. Parse with a SAX or streaming reader, process unit by unit, and memory stays flat.

The error

Locale QA agent: out of memory parsing 68MB XLIFF
run crashed before any QA check ran

Fix it

Step 1: Confirm the file size and structure

ls -lh big.xliff && grep -c "trans-unit" big.xliff | head -1

Expected: You see the size and unit count.

Step 2: Switch the parser to streaming

grep -n "parse\|DOMParser\|xml2js" scripts/qa-agent.js | head -5

Expected: You find the load-whole-file call to replace.

Step 3: Process unit by unit

node -e "console.log('pattern: sax stream, on trans-unit close run the QA checks, then discard the node')"

Expected: Memory stays flat regardless of file size.

Step 4: Re-run the QA checks

node scripts/qa-agent.js --file big.xliff | tail -3

Expected: The checks complete on the full file.

When to use this

• A locale QA agent crashes on a large XLIFF
• Out of memory during XML parsing

When NOT to use this

• The file is small but parsing is slow, check the parser
• The QA checks are wrong, fix the checks

Tool and version compatibility

• SAX/streaming XML parsers in Node
• XLIFF 1.2 trans-unit streaming

Variant phrasings

XLIFF 2.0 instead of 1.2

Same approach, stream on unit elements. The element name changes, the pattern does not.

need random access to units

Index first with a streaming pass, then seek. Two passes beat one OOM.

Why it happens

DOM parsers build the entire document tree in memory with heavy per-node overhead. A 68MB file easily needs over a gigabyte as a DOM, exceeding the agent's heap, while a streaming parser holds one unit at a time.

Edge cases

• Validate the XML separately with xmllint, streaming parsers are lenient
• Some checks need cross-unit context, buffer a window not the whole file
• Split giant files by unit count as a preprocessing step if streaming is hard

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GEnqcp7J5uCrwvpcrAZOXA