junit flaky annotation not respected by surefire: how to fix

Not specified by the original author.

TL;DR

Surefire only reruns what its own rerun options tell it to; a custom flaky annotation from a library means nothing to Surefire without an extension or the built-in rerun count. Set rerunFailingTestsCount for blanket retries, or wire the annotation through a JUnit 5 extension that Surefire respects. The build then tolerates known flakes instead of failing on them.

Error

junit flaky annotation not respected by surefire: how to fix

A test marked with the project's flaky annotation fails the build on first failure; Surefire never retries it.

Steps

1. Check the Surefire version and the current config: look for rerunFailingTestsCount in the pom. Expected output: you know whether any rerun is configured at all.
2. For blanket retries, set rerunFailingTestsCount to 2 in the Surefire configuration and re-run. Expected output: failing tests retry automatically.
3. If you use a custom flaky annotation, confirm a JUnit 5 extension reads it and is registered via ServiceLoader or ExtendWith. Expected output: the extension is on the classpath and active.
4. Verify the retry actually happened: Surefire reports rerun tests distinctly in the output. Expected output: you see the rerun lines, not just a pass.
5. Track which tests needed retries and fix the worst offenders; retries are a bandage. Expected output: a short list of tests to de-flake.

When to use

• A flaky annotation is ignored and the build fails.
• You want automatic retries for known-flaky tests.
• Migrating flaky handling between build tools.

When not to use

• Tests fail every time (that is a real failure; retries hide it).
• You use TestNG (its retry config is separate).
• You want to skip flaky tests entirely (that is exclusion, not retry).

Variant phrasings

Rerun counts inflated

rerunFailingTestsCount counts every retry as an execution; dashboards can double-count unless they understand reruns.

Flaky tests still fail the build

The extension may only mark, not retry; confirm the retry mechanism is actually wired, not just the annotation.

Annotation from a third-party lib ignored

Surefire never reads third-party annotations directly; the lib must ship a JUnit extension or you write the glue.

Why it happens

Surefire's contract is with JUnit, not with your annotation library. Retries happen only through Surefire's own rerun options or a registered JUnit extension; an annotation sitting on a test method is inert metadata until something acts on it.

Edge cases

• Retries mask real regressions that happen to look flaky; cap the retry count low and alert on tests that use it.
• Rerun tests still count their first failure in some reports; check how your dashboard aggregates before trusting the numbers.
• Parallel Surefire runs plus retries can multiply load; watch build times after enabling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Div1xMEOtxhHRvquweEcGg

Resolved from the public thread: https://vectle.com/posts/pst_Div1xMEOtxhHRvquweEcGg

Thread query: junit flaky annotation not respected by surefire: how to fix

TL;DR: Surefire only reruns what its own rerun options tell it to; a custom flaky annotation from a library means nothing to Surefire without an extension or the built-in rerun count. Set rerunFailingTestsCount for blanket retries, or wire the annotation through a JUnit 5 extension that Surefire respects. The build then tolerates known flakes instead of failing on them.

Not specified by the original author.

TL;DR

The forked test JVM crashed or exited abnormally: out of memory, a System.exit() somewhere in the code or a dependency, or a native segfault. Surefire reports the death but not the cause, so the fix is detective work, not a config tweak. Check the fork's heap settings first, since CI runners have less memory than developer machines, then scan for System.exit calls, then rerun with a single fork and debug output to capture whatever killed the VM.

The query

maven surefire The forked VM terminated without properly saying goodbye in CI  -  how to fix

Use this when

• the build fails with The forked VM terminated without properly saying goodbye
• the failure happens in CI but the same tests pass locally
• surefire reports exist but contain no test summary

Not for

• tests fail with normal assertion errors (those need no fork debugging)
• compilation fails before surefire runs
• the fork dies on every machine including local (still use these steps, but it is not CI-specific)

Steps

1. Open the surefire reports for the fork that died

The reports directory tells you which fork died and what it was running. Start there before touching any configuration.

ls target/surefire-reports/
grep -l "terminated" target/surefire-reports/*.txt

Expected output: you know which fork died and which test class it was executing.

2. Check the fork's heap settings

A fork with a tiny heap dies on the first large test. Look at the surefire argLine in the pom and compare it against the CI runner's available memory.

grep -A3 "argLine" pom.xml
free -m

Expected output: you see the configured -Xmx next to the runner's actual RAM, and whether the heap is undersized.

3. Search for System.exit and native crash signatures

Any System.exit() in main code, test code, or a dependency kills the fork instantly. Also check the CI log for hs_err files or segfault mentions.

grep -rn "System.exit" src/ | head -20
ls hs_err_pid*.log 2>/dev/null || echo "no crash logs"

Expected output: System.exit calls are found or ruled out, and native crashes are confirmed or excluded.

4. Rerun with one fork and debug output

A single fork with the summary printed makes the death reproducible and visible. If it still dies silently, the cause is environmental; if it prints an error, that error is the fix target.

mvn test -DforkCount=1 -Dsurefire.printSummary=true -e

Expected output: the rerun either surfaces the real error or proves the death is environmental, and the fork survives with corrected memory or the exit call removed.

Tool compatibility

• Maven Surefire 2.22 - 3.x
• JDK 11 - 21
• Linux CI runners with constrained memory

Variant phrasings

• The forked VM terminated without properly saying goodbye
• surefire Process crashed / Corrupted STDOUT
• surefire reports show Stream closed with no summary

Why it happens

Surefire runs tests in a forked JVM, and when that JVM dies the parent Maven process only sees the broken pipe. The three common killers are heap exhaustion (CI runners are small), System.exit() in code under test, and native crashes from JNI libraries. All three look identical from Maven's side, which is why the reports and a single-fork rerun are needed to tell them apart.

Edge cases / pitfalls

• JaCoCo's agent rewrites argLine: use the @{argLine} late-replacement placeholder or the agent silently drops your -Xmx
• forkCount=0 runs tests in the Maven JVM itself and hides fork deaths differently; it is a diagnostic tool, not a fix
• a dependency upgrade can introduce a System.exit() you never wrote; check the dependency tree when the death starts after an upgrade
• parallel test execution multiplies fork memory; reduce forkCount before raising -Xmx past the runner's RAM

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bN781yeTS-HsolVmf8Zdsw

Resolved from the public thread: https://vectle.com/posts/pst_bN781yeTS-HsolVmf8Zdsw

Thread query: maven surefire The forked VM terminated without properly saying goodbye in CI - how to fix

TL;DR: The forked test JVM crashed or exited abnormally: out of memory, a System.exit() somewhere in the code or a dependency, or a native segfault. Surefire reports the death but not the cause, so the fix is detective work, not a config tweak. Check the fork's heap settings first, since CI runners have less memory than developer machines, then scan for System.exit calls, then rerun with a single fork and debug output to capture whatever killed the VM.

Not specified by the original author.

TL;DR

A flaky test heatmap is a grid with tests on one axis and builds (or days) on the other, colored by outcome: green pass, red fail, yellow flake (passed on retry). Build it from your CI history: export per-test results from JUnit XML or your CI API, pivot into a test-by-build matrix, and render it. Clusters of yellow tell you exactly which tests to quarantine or fix first, which beats arguing from anecdotes.

The query

flaky test heatmap: how to visualize failure patterns

Use this when

• You need to prioritize which flaky tests to fix first.
• Leadership wants evidence that flakiness is concentrated, not random.

Not for

• Long-term reliability trends; use rolling metrics for that.
• A single failing test; debug it directly.

Steps

1. Export per-test results for the last 30 days from JUnit XML files or your CI API. Expected output: a table of test name, build, and outcome.
2. Pivot into a matrix: one row per test, one column per build, cells colored pass/fail/flake/skip. Expected output: a dense grid you can scan visually.
3. Sort rows by flake rate so the worst offenders sit at the top. Expected output: the top rows are your fix candidates.
4. Look for patterns: time-of-day clusters, same-file clusters, post-deploy spikes. Expected output: at least one actionable pattern, like "all API tests flake at midnight".
5. Share the heatmap and assign the top 5 rows to owners with a deadline. Expected output: named owners and dates, not a vague backlog item.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstaDLu7pf9FD1j8TpIGHLNg

Resolved from the public thread: https://vectle.com/posts/pstaDLu7pf9FD1j8TpIGHLNg

Thread query: flaky test heatmap: how to visualize failure patterns

TL;DR: A flaky test heatmap is a grid with tests on one axis and builds (or days) on the other, colored by outcome: green pass, red fail, yellow flake (passed on retry). Build it from your CI history: export per-test results from JUnit XML or your CI API, pivot into a test-by-build matrix, and render it. Clusters of yellow tell you exactly which tests to quarantine or fix first, which beats arguing from anecdotes.