agent's eval comparison used different seeds across runs: nondeterminism fix
Fixes nondeterministic eval comparisons by pinning random seeds, running each variant multiple times, and comparing distributions instead of single runs. Use when an agent's A/B eval conclusions flip between runs or seeds differ across compared runs. Not for deterministic unit-test flakes, which are a test-isolation problem rather than a seeding problem.
TL;DR
An eval comparison run with different seeds is not a comparison, it is two anecdotes. Pin one seed (or a fixed seed set) across all variants, run each variant N times, record the seed in the run metadata, and compare distributions. Any conclusion drawn from mismatched seeds is invalid.
Error
agent's eval comparison used different seeds across runs: nondeterminism fixVariant A beat variant B on Tuesday and lost on Wednesday because each run used a different random seed; the agent shipped the "winner" anyway.
Steps
- Audit the last comparison: extract the seed (or seed source) from each run's metadata. Expected output: proof the seeds differed, or proof no seed was recorded at all.
- Pin seeds: one fixed seed per comparison, or a fixed set of K seeds run for every variant. Expected output: a seed policy in the eval config.
- Re-run every variant under the pinned seeds, multiple trials each, and record seed plus trial in the metadata. Expected output: comparable numbers with provenance.
- Compare distributions (mean and spread across trials), not single best runs. Expected output: a winner that survives across trials, or an honest "no significant difference".
- Gate decisions on the comparison: no ship, no report, no agent claim without matched-seed results. Expected output: eval conclusions that reproduce.
When to use
- Eval conclusions flip between runs.
- Compared runs have different seeds or no recorded seed.
- An agent is choosing between prompts, models, or configs based on evals.
When not to use
- The tests are deterministic unit tests (fix isolation, not seeding).
- The nondeterminism comes from external services or timing, not RNG (pinning the seed will not help; control the environment instead).
- You are doing exploratory single runs to smoke-test a setup (no comparison being made).
Variant phrasings
Agent cherry-picked the best of five unseeded runs
Same fix: matched seeds plus a rule against reporting max-of-N as the headline number.
Eval harness defaults to time-based seeds
Time-based seeding guarantees mismatched comparisons. Override the default in the harness config.
Nonzero temperature makes "same seed" still nondeterministic
For LLM-output evals, pin the seed AND set temperature to 0 for comparisons, or budget enough trials to average out sampling noise.
Why it happens
Eval harnesses often seed from the clock by default, and agents rarely check. Each run is then a different draw, and with noisy tasks the winner is whoever got lucky. The agent reports the number in front of it, and nobody notices the comparison was never valid.
Edge cases
- Some libraries ignore the global seed and use their own RNG; audit every RNG source in the pipeline, not just the top-level seed.
- Parallel workers can consume RNG state in different orders; fix worker count as well as seed.
- GPU nondeterminism can survive seed pinning; for exact reproducibility, also pin the deterministic-ops flags the framework offers.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstqqzMULAfbqQwWCIQjwrfw