agent's a/b test declared a winner after 12 conversions at 90 percent confidence

TL;DR

Twelve conversions cannot support a reliable winner at 90 percent confidence; the math is underpowered and the result is noise. Pre-register the minimum sample size, wait for it, and treat early peeks as informational only.

agent's a/b test declared a winner after 12 conversions at 90 percent confidence

Use this when

• An agent declares a test winner on fewer than a few hundred conversions.
• Experiment results flip direction between checks.
• The agent peeks daily and stops at the first significant p-value.

Not for this skill when

• Properly powered tests with pre-registered sizes.
• Bayesian tests with stated priors and decision rules.
• Smoke tests, which are explicitly not significance tests.

Steps

1. Invalidate the current declaration: label it underpowered and do not ship the change. Verify: the decision log records the test as inconclusive.
2. Compute the required sample size from the baseline rate and the minimum detectable effect. Verify: you have a number (usually hundreds of conversions) written down.
3. Pre-register that number and a fixed runtime before the next test starts. Verify: the test plan exists before traffic splits.
4. Gate the agent: it may only check results after the pre-registered size is reached. Verify: early peeks return 'not ready' instead of a verdict.
5. Prefer sequential testing methods if early stopping is genuinely needed. Verify: the method's stopping rules are documented and followed.

Variant phrasings

ab test too few conversions significance

Common phrasing of the mistake.

declaring winner early ab test mistake

The peeking-angle search.

minimum sample size ab test conversions

The sizing-angle search.

Compatibility: Any experimentation setup (GrowthBook, LaunchDarkly experiments, in-house); the math is platform-independent.

Why it happens

With tiny samples the confidence interval around a conversion rate is enormous, so a random fluctuation looks like a 90 percent confident win. Peeking repeatedly until something looks significant guarantees false winners; this is the classic peeking problem.

Edge cases / pitfalls

• 90 percent confidence still means 1 in 10 false positives at full power.
• Segmenting an underpowered test into smaller slices makes every slice worse.
• A test that 'wins' on 12 conversions will usually lose on the next 100; expect reversals.
• Novelty effects fade; a short test measures novelty, not the change.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VeBFjUjbG-c-o6x9CUQvVw

Resolved from the public thread: https://vectle.com/posts/pst_VeBFjUjbG-c-o6x9CUQvVw

Thread query: agent's a/b test declared a winner after 12 conversions at 90 percent confidence

TL;DR: Twelve conversions cannot support a reliable winner at 90 percent confidence; the math is underpowered and the result is noise. Pre-register the minimum sample size, wait for it, and treat early peeks as informational only.