Exception in thread main java.lang.OutOfMemoryError GATK HaplotypeCaller fails:

Not specified by the original author.

Elasticsearch OutOfMemory on a small VPS: heap sizing and mlockall

TL;DR

[accepted answer]: this is a heap space issue. Rules of thumb: 1. Give Elasticsearch about half the box's RAM as heap, e.g.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

export ESHEAPSIZE=2g on a 4GB machine. Never set heap above ~50% of RAM (the OS needs the rest for filesystem cache), and never above ~32GB or you lose compressed object pointers. 2. Lock the memory for the JVM: set bootstrap.mlockall to true in the config so the heap never gets swapped out. 3. Investigate the disk: with only 30 small articles a 26GB data folder means something is off. Run GET cat/indices to find which index is huge (old log indexes that never got deleted are the usual suspect), and GET /nodes/stats for detailed node info. Tiny VPS + defaults is the classic combo: the default heap is small and shards/indexes pile up silently until the node falls over.

Context: Stack Overflow #34601697 (accepted answer, 17 votes): a small blog on a 4GB VPS runs fine for two months, then Elasticsearch dies with an Out of Memory (heap space) error. The asker also notices the data folder is 26GB for just 30 small articles.

Source: Stack Overflow #34601697 (accepted answer, 17 votes): a small blog on a 4GB VPS runs fine for two months, then Elasticsearch dies with an Out of Memory (heap space) error. The asker also notices the data folder is 26GB for just 30 small articles.

Verified resolution: [accepted answer]: this is a heap space issue. Rules of thumb: 1. Give Elasticsearch about half the box's RAM as heap, e.g. export ESHEAPSIZE=2g on a 4GB machine. Never set heap above ~50% of RAM (the OS needs the rest for filesystem cache), and never above ~32GB or you lose compressed object pointers. 2. Lock the memory for the JVM: set bootstrap.mlockall to true in the config so the heap never gets swapped out. 3. Investigate the disk: with only 30 small articles a 26GB data folder means something is off. Run GET cat/indices to find which index is huge (old log indexes that never got deleted are the usual suspect), and GET /nodes/stats for detailed node info. Tiny VPS + defaults is the classic combo: the default heap is small and shards/indexes pile up silently until the node falls over.

Source URL: https://stackoverflow.com/questions/34601697

Not specified by the original author.

TL;DR

For "couldn't allocate memory for bam_mem": from the man page: -m INT, Approximately the maximum required memory per thread, specified either in bytes or with a K, M, or G suffix. [768 MiB] So samtools sort -m 5G -@ 12 requests up to ~60 GiB (5 GiB × 12 threads), not 5 GiB. This is the single most common cause of Biostars "sort ran out of memory" threads: the declared -m looked reasonable, but multiplied by -@ it exceeded the machine or the job's memory limit.

couldn't allocate memory for bam_mem

When to use

You are seeing this: samtools sort -m is per thread, not total. Use this skill when you run into "Fix samtools sort memory failures and OOM kills".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

0. The one fact that explains most failures

samtools sort -m is per thread, not total. From the man page:

-m INT, Approximately the maximum required memory per thread, specified either in bytes or with a K, M, or G suffix. [768 MiB]

So samtools sort -m 5G -@ 12 requests up to ~60 GiB (5 GiB × 12 threads), not 5 GiB. This is the single most common cause of Biostars "sort ran out of memory" threads: the declared -m looked reasonable, but multiplied by -@ it exceeded the machine or the job's memory limit.

Compute the real request before touching anything:

total ≈ (-m value) × (-@ threads)

1. Classify the failure

A. samtools sort: couldn't allocate memory for bam_mem

The sort's per-thread allocation failed. Compare -m × -@ against actual available RAM (free -g):

• If -m × -@ exceeds free RAM (or the scheduler/Slurm/Nextflow memory limit for the job): over-requested memory. Fix: lower -m, lower -@, or both so the product fits. Example on a 32 GiB node:

  samtools sort -m 2G -@ 8 -T /scratch/sort_tmp -o sorted.bam in.bam
  # 2G x 8 threads = ~16 GiB, comfortably inside 32 GiB

• If -m × -@ fits easily yet allocation still fails: suspect the machine is actually smaller than you think (a VM resized to MB instead of GB is a real Biostars case) or another process holds the memory. Check free -g and dmesg | tail for the truth.

B. Shell prints Killed / exit code 137 / line N: [PID] Killed samtools sort

That's the Linux OOM killer (or the scheduler killing an over-limit job), not samtools failing cleanly. Same fix as A, the product -m × -@ is too big plus check the scheduler: in Nextflow/Slurm/SGE the job's declared memory (e.g. memory '32 GB') is the limit, not the node's RAM. Size the sort to the declared memory:

# nextflow process declaring 32 GB / 8 cpus:
samtools sort -m 3G -@ 8 -T ./sort_tmp -o out.bam in.bam
# 3G x 8 = ~24 GiB, under the 32 GB declaration with headroom

Splitting one process into two sequential sorts does not help an OOM: each sort's -m × -@ is what matters, not how many sorts ran.

C. No space left on device / sort stalls with growing temp files

When the data doesn't fit in -m × -@, sort spills to temp files named [tmpprefix].NNNN.bam in the -T directory, then merges them single-threaded at the end. Two sub-causes:

1. The -T directory is full or tiny. /tmp is often a tmpfs physical RAM, not disk, so -T /tmp on a big sort burns RAM twice (sort buffers and temp files) and then fails. Point -T at a real scratch filesystem with free space:

   mkdir -p /scratch/$USER/sort_tmp
   samtools sort -m 2G -@ 8 -T /scratch/$USER/sort_tmp -o sorted.bam in.bam

Check with df -h on the -T directory before the sort.

2. -m is far too small for the data, so sort creates thousands of temp files and the final merge crawls. Raise -m (keeping the product under available RAM) instead of lowering it. If you can't raise -m, accept the slow merge, lowering -m further makes it worse.

D. Name-sort (-n) pipelines OOM where coordinate sorts don't

samtools sort -n (name sort, used before fixmate/markdup) needs substantially more memory than a coordinate sort on the same file. If only the -n stage fails:

• Give the -n stage a bigger -m than the coordinate stage, or

• use samtools collate instead: the man page notes "Consider using samtools collate instead if you need name collated data without a full lexicographical sort", collate is designed for exactly the fixmate/markdup input pattern and uses less memory.

Also remember: -n (and -t) sorts cannot be indexed, samtools index requires coordinate sort. If a downstream tool complains about a missing index, the fix is a second coordinate sort, not re-running the name sort.

2. Rules of thumb

• Default -m is 768 MiB per thread. On a modern node, 4 8 threads at 1 2 GiB each (-m 2G -@ 4) sorts most BAMs fast; more threads help up to the disk's write bandwidth, then stop helping.

• -m has a floor of 1M (sort enforces it to avoid a temp-file explosion).

• Always set -T explicitly to a scratch dir with space; never rely on /tmp on shared systems.

• Verify the sorted result: samtools view -H sorted.bam | grep '^@HD' should show SO:coordinate (or SO:queryname for -n), and samtools quickcheck sorted.bam should exit silently.

Quick decision tree

1. couldn't allocate memory for bam_mem → compute -m × -@ vs free RAM → lower -m/-@.
2. Killed / exit 137 → same math, but against the job's declared memory (scheduler/Nextflow), not the node.
3. No space left on device or endless temp files → -T on real scratch disk (df -h first); don't use tmpfs /tmp.
4. Only -n fails → bigger -m for that stage, or samtools collate.
5. Merge phase slow but progressing → -m too small; raise it if RAM allows.

Built from the samtools sort man page (-m is approximately the maximum required memory per thread, default 768 MiB; -T temp-file behavior; -n incompatible with samtools index; samtools collate as the name-collation alternative) and recurring Biostars threads on sort memory failures in Snippy and Nextflow pipelines.

Not specified by the original author.

TL;DR

Gatling holds virtual-user state and response data in the heap, and the default JVM heap is sized for small runs. Raise the max heap, reduce what each user retains (especially response bodies), and confirm with GC logging. The simulation then survives its own peak.

Error

gatling simulation out of memory: heap tuning

Gatling simulation dies with java.lang.OutOfMemoryError: Java heap space at peak user injection.

Steps

1. Capture the OOM type from the crash: heap space versus metaspace point at different fixes. Expected output: you know whether to grow the heap or the metaspace.
2. Raise the max heap: set it in the Gatling launch config or JAVA_OPTS to a value the machine actually has. Expected output: the JVM starts with the larger heap.
3. Trim per-user retention: avoid storing full response bodies in checks or session attributes unless the scenario needs them. Expected output: heap per virtual user drops.
4. Re-run the simulation and watch GC logs for long pauses or a climbing old generation. Expected output: GC keeps up and the run completes.
5. If it still OOMs at peak, lower the peak injection rate or split the scenario across runs. Expected output: a load profile the heap can sustain.

When to use

• Gatling dies with OutOfMemoryError during a run.
• OOM happens only at peak user counts.
• You are sizing Gatling for a new load profile.

When not to use

• The application under test OOMs (tune the app, not Gatling).
• The simulation is slow but completes (that is tuning, not OOM).
• Metaspace errors on startup (that is classpath bloat, not heap).

Variant phrasings

OOM only at ramp peak

Peak concurrency is the memory high-water mark; size the heap for the peak, not the average.

Metaspace OOM

Too many classes loaded, often from huge feeders or scripts; raise MaxMetaspaceSize or trim the classpath.

Heap dump analysis

A dump at OOM shows what the heap held; look for session attributes and check bodies first.

Why it happens

Each virtual user carries session state, and checks can pin response bodies in memory until the scenario ends. Memory grows with concurrent users, so a profile that fits at 100 users can exhaust the default heap at 1000.

Edge cases

• Setting the heap larger than available RAM causes swapping, which looks like a hang rather than an OOM.
• G1GC is the sane default for Gatling; exotic collectors need retuning of pause targets.
• Feeder files load fully into memory; huge CSV feeders belong in streaming or batched mode.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstjsSktS6MSIcHvezCFIFIA

Resolved from the public thread: https://vectle.com/posts/pstjsSktS6MSIcHvezCFIFIA

Thread query: gatling simulation out of memory: heap tuning

TL;DR: Gatling holds virtual-user state and response data in the heap, and the default JVM heap is sized for small runs. Raise the max heap, reduce what each user retains (especially response bodies), and confirm with GC logging. The simulation then survives its own peak.