solr 'OutOfMemoryError' during a full reindex: how to batch the commits

solr 'OutOfMemoryError' during a full reindex: how to batch the commits

TL;DR

Commit in batches - send documents in chunks (e.g. 1k-10k per batch) with a commit at the end of each chunk, instead of one giant commit or a commit per document. Both extremes blow the heap: per-document commits drown in segment overhead, and one mega-commit holds everything in memory. Batched commits keep heap flat from the first document to the last.

The error

OutOfMemoryError

(in Solr logs during a full reindex; often Java heap space)

Steps

1. Confirm the failure mode in the Solr log: find the OutOfMemoryError and the indexing pattern around it (batch sizes, commit frequency).

   Expected: the OOM timestamp lines up with the reindex run, and you can see whether commits are per-document or end-of-run.

2. Restructure the indexer to send fixed-size batches. Accumulate documents client-side and POST each batch, with a commit after each batch:

# pseudo-flow for the indexer
batch = []
for doc in documents:
    batch.append(doc)
    if batch_has_5000_docs:
        post(batch); commit(); batch = []
post(batch); commit()  # final partial batch

Expected: heap usage stays roughly flat across the whole reindex instead of climbing to the OOM.

3. Tune the batch size empirically: start at 1000, watch GC and heap during a run, and raise toward 10000 only while heap stays flat. Big documents need smaller batches.

   Expected: a batch size where a full pass completes with heap never near the max.

4. Separate soft commits from hard commits during the bulk load: avoid autocommit-every-N-seconds churning segments mid-reindex; do explicit hard commits per batch and one final optimize-free commit at the end.

   Expected: fewer segments, less merge pressure, lower heap during the run.

5. Size the heap once for the workload (not per incident): enough for one batch plus merge headroom, and never heap plus OS cache above physical RAM.

   Expected: reindexes complete repeatably; the next data-growth doubling does not bring the OOM back.

Use this when

• Solr throws OutOfMemoryError during a full reindex.
• The reindex works on a small collection and dies as the collection grows.
• You are profiling or operating a Solr reindex pipeline.

Not for this skill when

• The OOM happens on queries, not indexing (that is query/filterCache sizing, a different problem).
• The reindex fails with disk-full errors (that is segment or disk capacity, not heap).
• You are on SolrCloud and the OOM is on ZooKeeper or an overseer node (different component).

Variant phrasings

solr full reindex java heap space

Same failure; batch the commits.

solr out of memory indexing large collection

Batch size plus commit strategy, as above.

solr reindex dies halfway with OOM

Halfway death is the heap climbing per document - batching flattens it.

Why it happens

Uncommitted documents accumulate in memory: with one giant end-of-run commit, the entire reindex sits in heap until the final commit flushes it. With per-document commits, each commit pays segment-creation overhead and the merge queue backs up, which also pins memory. Batching bounds both: only one batch is ever uncommitted, and commits are infrequent enough that segment merging keeps up.

Edge cases

• Do not run an optimize (forceMerge) after every batch; it is expensive and usually unnecessary - one at the very end, or none, since merges handle it.
• Near-real-time search during a reindex needs soft commits, but soft commits open new searchers per commit; during bulk load, prefer visibility at batch boundaries.
• If documents are huge (large stored fields), even 1000-doc batches can OOM; size batches by bytes, not just count.
• A reindex into a live collection doubles segment count temporarily; watch disk alongside heap.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstaa21ceKZSczmuBowBe99Q