agent's quarantine threshold was computed from 3 samples - one failure quarantined a healthy test

TL;DR

The agent quarantined a healthy test after one failure in three runs. Require a minimum sample size (e.g. 10+ runs) and a failure-rate threshold before quarantining; a single failure is a signal to investigate, not to quarantine. Re-run thin-evidence quarantines with proper sampling.

The symptom

agent's quarantine threshold was computed from 3 samples  -  one failure quarantined a healthy test

Steps

1. Audit the quarantine decision

Check how many runs the quarantine was based on. Three runs with one failure is noise, not evidence of flakiness. Expected: The sample count that drove the decision (3 runs).

2. Unquarantine and resample properly

Remove the quarantine flag and run the test 10 or more times. A healthy test passes consistently; a flaky one shows a real failure rate. Expected: 10+ clean runs proving the test is healthy.

3. Set a statistical quarantine policy

Teach the healer: quarantine requires at least N runs (10 is a sane floor) AND a failure rate above a threshold (e.g. 20 percent), or a confirmed non-deterministic root cause. Single failures get investigation, not quarantine. Expected: No quarantine on fewer than N samples, ever.

4. Log the evidence with every quarantine

Each quarantine record must include the sample size, failure count, and the runs it was based on, so thin decisions are visible in review. Expected: Quarantine log entries that show their own statistical basis.

When to use

• A test was quarantined after very few runs
• A single failure triggered quarantine
• Quarantine records lack sample-size evidence

When not to use

• The test has a long history of intermittent failures (quarantine may be right)
• The root cause is proven non-deterministic (quarantine with the cause attached)
• The policy is already statistical (look for a different misfire)

Why it happens

With 3 samples, a single failure is statistically meaningless; even a 99-percent-reliable test fails once in a while. Quarantining on thin evidence removes healthy tests from the suite, shrinking coverage while the 'flakiness' dashboard fills with false positives.

Edge cases

• Expensive tests where 10 runs is costly: use fewer runs but require a reproduced root cause, not a bare failure.
• Newly added tests with no history: default to investigation, never auto-quarantine.
• Intermittent infrastructure noise: exclude infra-failed runs from the sample count.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_J63pW2MAyydpsZsGkHd3Sg