Skip to content
← Back to activity

Immutable solution version

Amazing AB test result? Check for bugs first, and always set triggering

Version 1Sign in to save

Complete solution

When an AB test shows an amazing result, assume a bug first: verify tracking, the feature, and the experiment setup before celebrating (Twyman's Law).

When an AB test shows an amazing result, assume a bug first: verify tracking, the feature, and the experiment setup before celebrating (Twyman's Law). And always define triggering, evaluate only the users who could actually see the change. Including unaffected visitors dilutes the effect, drags out runtime, and can make a real win look like noise.

Context: Web (dev.to, 5 most common AB testing mistakes): describes two classic traps. Twyman's Law: any figure that looks interesting or different is usually wrong, so an amazing early result usually means a bug in the feature, the tracking, or the experiment design, and you must double-check before presenting it. No triggering: if a feature only affects registered users but you evaluate all visitors including anonymous ones, the dilution kills sensitivity and can make the test impossible to evaluate.

Evidence

Source: Web (dev.to, 5 most common AB testing mistakes): describes two classic traps. Twyman's Law: any figure that looks interesting or different is usually wrong, so an amazing early result usually means a bug in the feature, the tracking, or the experiment design, and you must double-check before presenting it. No triggering: if a feature only affects registered users but you evaluate all visitors including anonymous ones, the dilution kills sensitivity and can make the test impossible to evaluate.

Verified resolution: When an AB test shows an amazing result, assume a bug first: verify tracking, the feature, and the experiment setup before celebrating (Twyman's Law). And always define triggering, evaluate only the users who could actually see the change. Including unaffected visitors dilutes the effect, drags out runtime, and can make a real win look like noise.

Source URL: https://DEV.to/daelmaak/5-most-common-ab-testing-mistakes-4bk1

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_wun00JpS1bsxFAGjg2LwOA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Amazing AB test result? Check for bugs first, and always set triggering <span>— Vectle</span></a>