agent's A/B test script assigned every visitor to control after cookie parse failed

TL;DR

Replace cookie-based bucketing with a deterministic hash of a stable visitor id, and on any parse failure fall back to the hash, never to a hardcoded arm. Visitors that cannot be identified get excluded from the experiment instead of forced into control. Assignment stays balanced even when cookies break.

agent's A/B test script assigned every visitor to control after cookie parse failed

Use this when

• an experiment shows 100 percent of traffic in one arm
• cookie parsing throws or returns empty for some visitors
• assignment depends on a value the client may not send
• you need bucketing that survives missing or corrupt cookies

Not for this skill when

• the experiment design or sample size is the problem
• you need help reading test results or significance
• traffic splitting is intentionally uneven by design

Steps

1. Replace cookie-based bucketing with a deterministic hash of a stable visitor id, modulo 100, compared against the variant split. Expected: assignment works with zero cookies and never collapses to one arm.
2. Validate the cookie parse and on failure assign by the hash fallback, never by a hardcoded arm. Expected: parse failures distribute evenly instead of landing 100 percent in control.
3. Log every assignment failure with the reason so a broken parser is visible in hours, not weeks. Expected: a dashboard alert on parse-failure rate.
4. Exclude visitors that cannot be identified from the experiment instead of forcing them into control. Expected: cleaner data and no bias toward either arm.
5. Add a startup self-test that assigns a thousand synthetic ids and asserts the split is within tolerance. Expected: a broken bucketing change fails the test before it ships.

Variant phrasings

experiment assigned everyone to control

Short version: hash a stable visitor id for bucketing so cookie failures cannot bias the split.

cookie parse failure broke variant assignment

Short version: fall back to deterministic hashing on parse failure, never a hardcoded arm.

A/B test collapsed to a single arm

Short version: validate the parse, log failures, and exclude unidentifiable visitors.

Compatibility: works with any feature-flag or experiment setup (LaunchDarkly, custom scripts, edge workers); the hashing is plain code with no library pinned.

Why it happens

The script read the variant from a cookie, and when parsing failed it hit a fallback branch that returned control for everyone. The failure was silent, so the experiment ran for days with no variance. One unguarded fallback turned a parsing bug into useless data.

Edge cases / pitfalls

• Hashing needs a stable id; if the id itself comes from the broken cookie, fix identity first.
• Excluding visitors changes the analyzed population; document the exclusion rule with the results.
• A modulo bias appears with tiny samples; the startup self-test catches it.
• Do not re-bucket mid-experiment when fixing this; start a new experiment so the data stays clean.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ka6Mwcennr4t3f0IffFYAQ