# solr 'OutOfMemoryError' during a full reindex: how to batch the commits

## TL;DR
Commit in batches - send documents in chunks (e.g. 1k-10k per batch) with a commit at the end of each chunk, instead of one giant commit or a commit per document. Both extremes blow the heap: per-document commits drown in segment overhead, and one mega-commit holds everything in memory. Batched commits keep heap flat from the first document to the last.

## The error

```text
OutOfMemoryError
```

   (in Solr logs during a full reindex; often `Java heap space`)

## Steps

1. Confirm the failure mode in the Solr log: find the OutOfMemoryError and the indexing pattern around it (batch sizes, commit frequency).

   Expected: the OOM timestamp lines up with the reindex run, and you can see whether commits are per-document or end-of-run.

2. Restructure the indexer to send fixed-size batches. Accumulate documents client-side and POST each batch, with a commit after each batch:

```
# pseudo-flow for the indexer
batch = []
for doc in documents:
    batch.append(doc)
    if batch_has_5000_docs:
        post(batch); commit(); batch = []
post(batch); commit()  # final partial batch
```

   Expected: heap usage stays roughly flat across the whole reindex instead of climbing to the OOM.

3. Tune the batch size empirically: start at 1000, watch GC and heap during a run, and raise toward 10000 only while heap stays flat. Big documents need smaller batches.

   Expected: a batch size where a full pass completes with heap never near the max.

4. Separate soft commits from hard commits during the bulk load: avoid autocommit-every-N-seconds churning segments mid-reindex; do explicit hard commits per batch and one final optimize-free commit at the end.

   Expected: fewer segments, less merge pressure, lower heap during the run.

5. Size the heap once for the workload (not per incident): enough for one batch plus merge headroom, and never heap plus OS cache above physical RAM.

   Expected: reindexes complete repeatably; the next data-growth doubling does not bring the OOM back.

## Use this when

- Solr throws OutOfMemoryError during a full reindex.
- The reindex works on a small collection and dies as the collection grows.
- You are profiling or operating a Solr reindex pipeline.

## Not for this skill when

- The OOM happens on queries, not indexing (that is query/filterCache sizing, a different problem).
- The reindex fails with disk-full errors (that is segment or disk capacity, not heap).
- You are on SolrCloud and the OOM is on ZooKeeper or an overseer node (different component).

## Variant phrasings

### solr full reindex java heap space
Same failure; batch the commits.

### solr out of memory indexing large collection
Batch size plus commit strategy, as above.

### solr reindex dies halfway with OOM
Halfway death is the heap climbing per document - batching flattens it.

## Why it happens
Uncommitted documents accumulate in memory: with one giant end-of-run commit, the entire reindex sits in heap until the final commit flushes it. With per-document commits, each commit pays segment-creation overhead and the merge queue backs up, which also pins memory. Batching bounds both: only one batch is ever uncommitted, and commits are infrequent enough that segment merging keeps up.

## Edge cases

- Do not run an optimize (forceMerge) after every batch; it is expensive and usually unnecessary - one at the very end, or none, since merges handle it.
- Near-real-time search during a reindex needs soft commits, but soft commits open new searchers per commit; during bulk load, prefer visibility at batch boundaries.
- If documents are huge (large stored fields), even 1000-doc batches can OOM; size batches by bytes, not just count.
- A reindex into a live collection doubles segment count temporarily; watch disk alongside heap.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aa21ceKZS_czmuBowBe99Q