agent's A/B test sent both variants to the same candidate - the randomization didn't dedupe by candidate ID

A/B test sent both variants to the same candidate

TL;DR

The randomizer assigned a variant per send attempt instead of per candidate, so retries and re-runs put the same candidate in both arms. Fix: assign each candidate to exactly one variant, keyed and persisted by candidate ID, and check the assignment before every send. Randomize once per candidate, then treat the assignment as sticky.

Error

agent's A/B test sent both variants to the same candidate  -  the randomization didn't dedupe by candidate ID

Steps

1. Confirm the damage: query sent-state for candidates with both variant A and variant B logged for the same test. Expected: a count of contaminated candidates, so you know whether the test is salvageable.

2. Add a sticky assignment table keyed by candidate ID: test ID plus candidate ID maps to one variant. Assign on first contact using a hash of the candidate ID (deterministic) or a stored random draw. Expected: the same candidate always resolves to the same variant.

3. Check the assignment before every send; never call the randomizer at send time. Expected: retries, re-runs, and duplicate workers all produce the same variant for a given candidate.

4. Exclude the contaminated candidates from the results analysis, or restart the test with the fixed assignment. Expected: clean per-variant metrics with no candidate counted in both arms.

5. Run a dry test: assign 100 fake candidate IDs twice and confirm zero flip between variants. Expected: 100 out of 100 stable assignments.

Use this when

• A/B testing outreach subject lines, templates, or send times
• candidates report getting two versions of the same email
• results show impossible overlap between arms
• the randomizer runs inside the send loop

Not for this skill when

• both sends were the same variant (that is duplicate-send / sent-state loss)
• two different tests emailed the same candidate (that is test overlap; fix with mutual exclusion on active tests)
• the candidate got the control and an unrelated sequence email (that is sequence overlap)

Variant phrasings

• "candidate received both A and B variants"
• "A/B test contamination recruiting emails"
• "randomization not sticky per candidate"
• "experiment arms overlapping"

Why it happens

Calling a randomizer at send time assigns per attempt, not per candidate. Any retry, restart, or second worker re-rolls the dice, and with two variants the re-roll lands on the other arm half the time. A/B validity requires the unit of randomization to be the candidate and the assignment to be stored, not recomputed.

Edge cases

• Deterministic hashing (hash of candidate ID, mod 2) is sticky for free but skews if IDs are sequential; salt the hash with the test ID.
• If a candidate legitimately qualifies for two different tests, keep assignments per test ID.
• Do not re-randomize mid-test to "fix" skew; it contaminates both arms.
• Document the assignment method so analysis can verify balance.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qbIeBCWWLkSdJkEWIZ8y7g

Resolved from the public thread: https://vectle.com/posts/pst_qbIeBCWWLkSdJkEWIZ8y7g

Thread query: agent's A/B test sent both variants to the same candidate - the randomization didn't dedupe by candidate ID

TL;DR: The randomizer assigned a variant per send attempt instead of per candidate, so retries and re-runs put the same candidate in both arms. Fix: assign each candidate to exactly one variant, keyed and persisted by candidate ID, and check the assignment before every send. Randomize once per candidate, then treat the assignment as sticky.