## TL;DR
Twelve conversions cannot support a reliable winner at 90 percent confidence; the math is underpowered and the result is noise. Pre-register the minimum sample size, wait for it, and treat early peeks as informational only.

```text
agent's a/b test declared a winner after 12 conversions at 90 percent confidence
```

## Use this when
- An agent declares a test winner on fewer than a few hundred conversions.
- Experiment results flip direction between checks.
- The agent peeks daily and stops at the first significant p-value.

## Not for this skill when
- Properly powered tests with pre-registered sizes.
- Bayesian tests with stated priors and decision rules.
- Smoke tests, which are explicitly not significance tests.

## Steps
1. Invalidate the current declaration: label it underpowered and do not ship the change. Verify: the decision log records the test as inconclusive.
2. Compute the required sample size from the baseline rate and the minimum detectable effect. Verify: you have a number (usually hundreds of conversions) written down.
3. Pre-register that number and a fixed runtime before the next test starts. Verify: the test plan exists before traffic splits.
4. Gate the agent: it may only check results after the pre-registered size is reached. Verify: early peeks return 'not ready' instead of a verdict.
5. Prefer sequential testing methods if early stopping is genuinely needed. Verify: the method's stopping rules are documented and followed.

## Variant phrasings
### ab test too few conversions significance
Common phrasing of the mistake.
### declaring winner early ab test mistake
The peeking-angle search.
### minimum sample size ab test conversions
The sizing-angle search.

Compatibility: Any experimentation setup (GrowthBook, LaunchDarkly experiments, in-house); the math is platform-independent.

## Why it happens
With tiny samples the confidence interval around a conversion rate is enormous, so a random fluctuation looks like a 90 percent confident win. Peeking repeatedly until something looks significant guarantees false winners; this is the classic peeking problem.

## Edge cases / pitfalls
- 90 percent confidence still means 1 in 10 false positives at full power.
- Segmenting an underpowered test into smaller slices makes every slice worse.
- A test that 'wins' on 12 conversions will usually lose on the next 100; expect reversals.
- Novelty effects fade; a short test measures novelty, not the change.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VeBFjUjbG-c-o6x9CUQvVw