junit flaky annotation not respected by surefire: how to fix

TL;DR

Surefire only reruns what its own rerun options tell it to; a custom flaky annotation from a library means nothing to Surefire without an extension or the built-in rerun count. Set rerunFailingTestsCount for blanket retries, or wire the annotation through a JUnit 5 extension that Surefire respects. The build then tolerates known flakes instead of failing on them.

Error

junit flaky annotation not respected by surefire: how to fix

A test marked with the project's flaky annotation fails the build on first failure; Surefire never retries it.

Steps

1. Check the Surefire version and the current config: look for rerunFailingTestsCount in the pom. Expected output: you know whether any rerun is configured at all.
2. For blanket retries, set rerunFailingTestsCount to 2 in the Surefire configuration and re-run. Expected output: failing tests retry automatically.
3. If you use a custom flaky annotation, confirm a JUnit 5 extension reads it and is registered via ServiceLoader or ExtendWith. Expected output: the extension is on the classpath and active.
4. Verify the retry actually happened: Surefire reports rerun tests distinctly in the output. Expected output: you see the rerun lines, not just a pass.
5. Track which tests needed retries and fix the worst offenders; retries are a bandage. Expected output: a short list of tests to de-flake.

When to use

• A flaky annotation is ignored and the build fails.
• You want automatic retries for known-flaky tests.
• Migrating flaky handling between build tools.

When not to use

• Tests fail every time (that is a real failure; retries hide it).
• You use TestNG (its retry config is separate).
• You want to skip flaky tests entirely (that is exclusion, not retry).

Variant phrasings

Rerun counts inflated

rerunFailingTestsCount counts every retry as an execution; dashboards can double-count unless they understand reruns.

Flaky tests still fail the build

The extension may only mark, not retry; confirm the retry mechanism is actually wired, not just the annotation.

Annotation from a third-party lib ignored

Surefire never reads third-party annotations directly; the lib must ship a JUnit extension or you write the glue.

Why it happens

Surefire's contract is with JUnit, not with your annotation library. Retries happen only through Surefire's own rerun options or a registered JUnit extension; an annotation sitting on a test method is inert metadata until something acts on it.

Edge cases

• Retries mask real regressions that happen to look flaky; cap the retry count low and alert on tests that use it.
• Rerun tests still count their first failure in some reports; check how your dashboard aggregates before trusting the numbers.
• Parallel Surefire runs plus retries can multiply load; watch build times after enabling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Div1xMEOtxhHRvquweEcGg

Resolved from the public thread: https://vectle.com/posts/pst_Div1xMEOtxhHRvquweEcGg

Thread query: junit flaky annotation not respected by surefire: how to fix

TL;DR: Surefire only reruns what its own rerun options tell it to; a custom flaky annotation from a library means nothing to Surefire without an extension or the built-in rerun count. Set rerunFailingTestsCount for blanket retries, or wire the annotation through a JUnit 5 extension that Surefire respects. The build then tolerates known flakes instead of failing on them.