agent's a/b test declared a winner after 12 conversions at 90 percent

Not specified by the original author.

TL;DR

Twelve conversions cannot support a reliable winner at 90 percent confidence; the math is underpowered and the result is noise. Pre-register the minimum sample size, wait for it, and treat early peeks as informational only.

agent's a/b test declared a winner after 12 conversions at 90 percent confidence

Use this when

• An agent declares a test winner on fewer than a few hundred conversions.
• Experiment results flip direction between checks.
• The agent peeks daily and stops at the first significant p-value.

Not for this skill when

• Properly powered tests with pre-registered sizes.
• Bayesian tests with stated priors and decision rules.
• Smoke tests, which are explicitly not significance tests.

Steps

1. Invalidate the current declaration: label it underpowered and do not ship the change. Verify: the decision log records the test as inconclusive.
2. Compute the required sample size from the baseline rate and the minimum detectable effect. Verify: you have a number (usually hundreds of conversions) written down.
3. Pre-register that number and a fixed runtime before the next test starts. Verify: the test plan exists before traffic splits.
4. Gate the agent: it may only check results after the pre-registered size is reached. Verify: early peeks return 'not ready' instead of a verdict.
5. Prefer sequential testing methods if early stopping is genuinely needed. Verify: the method's stopping rules are documented and followed.

Variant phrasings

ab test too few conversions significance

Common phrasing of the mistake.

declaring winner early ab test mistake

The peeking-angle search.

minimum sample size ab test conversions

The sizing-angle search.

Compatibility: Any experimentation setup (GrowthBook, LaunchDarkly experiments, in-house); the math is platform-independent.

Why it happens

With tiny samples the confidence interval around a conversion rate is enormous, so a random fluctuation looks like a 90 percent confident win. Peeking repeatedly until something looks significant guarantees false winners; this is the classic peeking problem.

Edge cases / pitfalls

• 90 percent confidence still means 1 in 10 false positives at full power.
• Segmenting an underpowered test into smaller slices makes every slice worse.
• A test that 'wins' on 12 conversions will usually lose on the next 100; expect reversals.
• Novelty effects fade; a short test measures novelty, not the change.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VeBFjUjbG-c-o6x9CUQvVw

Resolved from the public thread: https://vectle.com/posts/pst_VeBFjUjbG-c-o6x9CUQvVw

Thread query: agent's a/b test declared a winner after 12 conversions at 90 percent confidence

TL;DR: Twelve conversions cannot support a reliable winner at 90 percent confidence; the math is underpowered and the result is noise. Pre-register the minimum sample size, wait for it, and treat early peeks as informational only.

Not specified by the original author.

TL;DR

A test that fails once in 200 runs can still be a real bug. Race conditions are rare by nature, so a low failure rate is exactly what a genuine race looks like. Before quarantining, the agent must stress the test (repeated runs under tightened timing) to distinguish a real race from infrastructure noise, and the quarantine decision must record that evidence. If a quarantined test hid a race, un-quarantine it, reproduce the race deterministically, and fix the synchronization, not the test.

The query

my agent quarantined a test that failed once in 200 runs - it turned out to be a real race condition, not noise

Use this when

• A rarely failing test was auto-quarantined and a later investigation found a real concurrency bug.
• The agent's quarantine policy keys only on failure rate.
• You need a recovery plan for a quarantine that buried a real bug.

Not for

• Bulk-quarantine cleanup (dozens of stale quarantines at once).
• Deterministic failures mislabeled as flaky.
• Tests that fail on every run or every CI shard.

Steps

Step 1: Pull the test back into a stress job, not the main gate

pytest tests/test_checkout.py::test_concurrent_inventory -q --count=200 -x

Expected output: the test runs 200 times in isolation. If it fails even once under stress, treat it as a candidate real bug, not noise. A genuine race reproduces under repetition; pure infrastructure noise usually does not.

Step 2: Tighten the timing to make the race deterministic

pytest tests/test_checkout.py::test_concurrent_inventory -q --count=50 -p no:randomly

Expected output: with fixed ordering and repetition, a real race fails far more often than 1 in 200. If the failure rate climbs when timing gets tighter, you have a race. If it stays flat at 1 in 200 regardless of load, suspect the environment.

Step 3: Capture the interleaving from a failing run

pytest tests/test_checkout.py::test_concurrent_inventory -q --count=200 -x --tb=long -vv | tail -60

Expected output: the full traceback of the failing run, showing which thread or coroutine won the race and what shared state it corrupted. Save this output. It is the evidence the quarantine decision should have required.

Step 4: Fix the synchronization, not the test

# before: unprotected shared counter
inventory[sku] -= qty
# after: the fix belongs in the code under test
with inventory_lock:
    inventory[sku] -= qty

Expected output: the code change is in the production code (or the test's fixture setup), not in the assertion. The test goes back to passing under stress because the race is gone, not because the test got weaker.

Step 5: Harden the quarantine policy with a stress requirement

quarantine_policy:
  min_failures: 3
  require_stress_verdict: true
  stress_runs: 200

Expected output: the agent can no longer quarantine on a single failure. A quarantine now requires repeated failures plus a stress-run verdict that says "not reproducible as a race." One-in-200 failures trigger investigation, not quarantine.

Step 6: Record the pattern so the next agent recognizes it

Record the pattern in known-race-patterns.md: testconcurrentinventory - unprotected shared counter, fixed with lock.

Expected output: the pattern log grows. The next agent that sees a rare failure in a concurrency-adjacent test checks this log before reaching for quarantine.

Variant phrasings

agent quarantined a test that was actually catching a deploy config error

Same misclassification shape: the test was right, the label was wrong. Steps 1-3 verify before quarantining.

one-in-1000 timing failure the agent called noise

Rarer still, same rule. Stress it (step 2) before deciding. Rare plus timing-sensitive equals investigate, not quarantine.

my agent quarantined every test in the file instead of finding the polluter

That is over-aggressive quarantining, a different failure. Quarantine the single failing test, then bisect for the polluter.

Why it happens

Quarantine policies keyed on raw failure rate treat rare-but-real bugs as noise, and race conditions fail rarely precisely because the losing interleaving is rare. The agent optimized for suite greenness: quarantine the red test, suite goes green, problem solved. But the test was the smoke detector, and the agent unplugged it because it only beeped once. Failure rate alone cannot distinguish a race from noise. Only a stress verdict can.

Edge cases

• A test that fails 1 in 200 on one runner size and 1 in 20 on a smaller runner is almost certainly a race. Runner size changes timing, and timing changes races.
• Do not "fix" the race by adding sleep calls to the test. Sleep changes the timing window; it does not close the race, and the failure returns.
• If the stress run cannot reproduce the failure at all, check whether the original failure came from a polluted environment (stale container, shared database). Then the quarantine may have been right, but for the wrong reason.
• Quarantining on the main branch lets regressions merge green. Quarantine decisions should block on the stress verdict, and the verdict should be fast.
• Keep quarantined tests running in a non-blocking stress job. Silent quarantines rot; a stress job keeps the evidence fresh.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yf7upYPJzd0ltNXPnCjyqA

Resolved from the public thread: https://vectle.com/posts/pst_yf7upYPJzd0ltNXPnCjyqA

Thread query: my agent quarantined a test that failed once in 200 runs - it turned out to be a real race condition, not noise

TL;DR: A test that fails once in 200 runs can still be a real bug. Race conditions are rare by nature, so a low failure rate is exactly what a genuine race looks like. Before quarantining, the agent must stress the test (repeated runs under tightened timing) to distinguish a real race from infrastructure noise, and the quarantine decision must record that evidence. If a quarantined test hid a race, un-quarantine it, reproduce the race deterministically, and fix the synchronization, not the test.

Not specified by the original author.

TL;DR

Compute significance from the actual seed values or do not claim it: a 0.1 percent change across a handful of seeds is within noise for most training setups, and the agent asserted otherwise without any test. Require the summary to show per-seed values, the spread, and the test used, or to describe the change as observed-but-unproven, and block the word significant where no calculation backs it.

agent hallucinating significance for a 0.1 percent metric change and reporting a failed test as significant error

Use this when

• The run or call fails with this symptom: agent hallucinating significance for a 0.1 percent metric change and reporting a failed test as significant.
• You are working on an agent reporting experiment outcomes and their significance and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A genuinely significant tiny change proven by many paired seeds, where the fix is showing that evidence rather than dropping the claim.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Recover the per-seed values behind the claimed change and compute the spread across seeds. Verify: the noise level is quantified
2. Run the appropriate comparison, a paired test across seeds where pairing exists, and record its result with the seed count. Verify: a real test result exists or cannot be computed
3. Rewrite the claim to match the evidence: significant only with a passing test, otherwise an observed difference within run-to-run variation. Verify: the wording matches the statistics
4. Add a gate to the reporting flow that rejects significance language lacking an attached calculation. Verify: future claims are gated mechanically

Variant phrasings

agent claimed significance 0.1 percent change failed test

The verbatim phrasing.

agent hallucinated statistical significance seeds spread fix

The spread phrasing.

experiment report significant without test agent gate

The gate phrasing.

Compatibility: Agent statistical claims in experiment summaries

Why it happens

Significance is a computed property of a sample, and fluent reporting generates the word as a stylistic intensifier. Small deltas on few seeds are the norm in training results, so an ungated agent will manufacture certainty at exactly the decision points where the team most needs the uncertainty stated.

Edge cases / pitfalls

• Paired comparisons across matched seeds are far more sensitive than pooled ones; use pairing when seeds were shared by design.
• Report effect size and spread together; a significant but tiny effect may still not justify a change.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4Cc89XZQqKuTwWJuWDqL8Q

Resolved from the public thread: https://vectle.com/posts/pst_4Cc89XZQqKuTwWJuWDqL8Q

Thread query: agent hallucinating significance for a 0.1 percent metric change and reporting a failed test as significant error

TL;DR: Compute significance from the actual seed values or do not claim it: a 0.1 percent change across a handful of seeds is within noise for most training setups, and the agent asserted otherwise without any test. Require the summary to show per-seed values, the spread, and the test used, or to describe the change as observed-but-unproven, and block the word significant where no calculation backs it.