agent compared profiled vs unprofiled runs as an A/B test and shipped the 'faster' variant -- which was just the...

TL;DR

You A/B tested a profiled build against an unprofiled build, and the "winner" was just the variant with the measurement turned off. A benchmark arm is only comparable when the instrumentation is identical. Re-run with both arms profiled or both arms clean. The honest result is usually a tie, plus a separately measured profiler cost.

agent compared profiled vs unprofiled runs as an A/B test and shipped the 'faster' variant -- which was just the uninstrumented one

1. Audit the test setup. Confirm what differed between arm A and arm B: code change, profiler attached, sample rate, agent enabled. Expected: the arms differed in instrumentation, not just in code.
2. Re-run with identical instrumentation. Put the profiler or agent on both arms at the same settings, or strip it from both, and re-run the traffic split. Expected: the performance gap collapses to near zero.
3. Measure the overhead as its own number. Run the same arm with the profiler on and off and record the delta. Expected: the delta explains the original "win" almost exactly.
4. Fix the process. Make identical instrumentation a required checklist item for every benchmark, and if the uninstrumented variant already shipped, verify it is actually equivalent code before deciding whether to keep it. Expected: no future A/B ships a winner that only won by being unmeasured.

Use this when

• The winning variant in a benchmark differs from the loser by instrumentation, not code
• A "faster" deploy coincides with an agent or profiler being removed
• The two arms were measured with different tooling

Not for this skill when

• Both arms were instrumented identically and one still wins: that is a real result, ship it
• The code actually differs between arms: then re-test with matched instrumentation to isolate the code effect

Variant phrasings

• "AB test profiled vs unprofiled variant misleading"
• "benchmark arms had different APM settings"
• "shipped faster variant that was just uninstrumented"

Why it happens

An A/B test attributes the outcome difference to the intended variable, the code change. But measurement cost is a second variable, and when it differs between arms it contributes its full size to the observed difference. The test then confidently recommends the arm with less measurement, which is a statement about the tooling, not the code.

Edge cases

• Canary analysis has the same trap: canary with the profiler on versus baseline with it off
• Feature flags that toggle instrumentation per arm recreate the problem silently
• Even "identical" agents can differ by version or config between arms: diff the agent configs too

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gMf4e52nzW9ZbFyP4vbq1w