# A/B test sent both variants to the same candidate

## TL;DR
The randomizer assigned a variant per send attempt instead of per candidate, so retries and re-runs put the same candidate in both arms. Fix: assign each candidate to exactly one variant, keyed and persisted by candidate ID, and check the assignment before every send. Randomize once per candidate, then treat the assignment as sticky.

## Error
```text
agent's A/B test sent both variants to the same candidate  -  the randomization didn't dedupe by candidate ID
```

## Steps
1. Confirm the damage: query sent-state for candidates with both variant A and variant B logged for the same test.
   Expected: a count of contaminated candidates, so you know whether the test is salvageable.
2. Add a sticky assignment table keyed by candidate ID: test ID plus candidate ID maps to one variant. Assign on first contact using a hash of the candidate ID (deterministic) or a stored random draw.
   Expected: the same candidate always resolves to the same variant.
3. Check the assignment before every send; never call the randomizer at send time.
   Expected: retries, re-runs, and duplicate workers all produce the same variant for a given candidate.
4. Exclude the contaminated candidates from the results analysis, or restart the test with the fixed assignment.
   Expected: clean per-variant metrics with no candidate counted in both arms.
5. Run a dry test: assign 100 fake candidate IDs twice and confirm zero flip between variants.
   Expected: 100 out of 100 stable assignments.

## Use this when
- A/B testing outreach subject lines, templates, or send times
- candidates report getting two versions of the same email
- results show impossible overlap between arms
- the randomizer runs inside the send loop

## Not for this skill when
- both sends were the same variant (that is duplicate-send / sent-state loss)
- two different tests emailed the same candidate (that is test overlap; fix with mutual exclusion on active tests)
- the candidate got the control and an unrelated sequence email (that is sequence overlap)

## Variant phrasings
- "candidate received both A and B variants"
- "A/B test contamination recruiting emails"
- "randomization not sticky per candidate"
- "experiment arms overlapping"

## Why it happens
Calling a randomizer at send time assigns per attempt, not per candidate. Any retry, restart, or second worker re-rolls the dice, and with two variants the re-roll lands on the other arm half the time. A/B validity requires the unit of randomization to be the candidate and the assignment to be stored, not recomputed.

## Edge cases
- Deterministic hashing (hash of candidate ID, mod 2) is sticky for free but skews if IDs are sequential; salt the hash with the test ID.
- If a candidate legitimately qualifies for two different tests, keep assignments per test ID.
- Do not re-randomize mid-test to "fix" skew; it contaminates both arms.
- Document the assignment method so analysis can verify balance.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qbIeBCWWLkSdJkEWIZ8y7g