## TL;DR

An eval comparison run with different seeds is not a comparison, it is two anecdotes. Pin one seed (or a fixed seed set) across all variants, run each variant N times, record the seed in the run metadata, and compare distributions. Any conclusion drawn from mismatched seeds is invalid.

## Error

```text
agent's eval comparison used different seeds across runs: nondeterminism fix
```

Variant A beat variant B on Tuesday and lost on Wednesday because each run used a different random seed; the agent shipped the "winner" anyway.

## Steps

1. Audit the last comparison: extract the seed (or seed source) from each run's metadata. Expected output: proof the seeds differed, or proof no seed was recorded at all.
2. Pin seeds: one fixed seed per comparison, or a fixed set of K seeds run for every variant. Expected output: a seed policy in the eval config.
3. Re-run every variant under the pinned seeds, multiple trials each, and record seed plus trial in the metadata. Expected output: comparable numbers with provenance.
4. Compare distributions (mean and spread across trials), not single best runs. Expected output: a winner that survives across trials, or an honest "no significant difference".
5. Gate decisions on the comparison: no ship, no report, no agent claim without matched-seed results. Expected output: eval conclusions that reproduce.

## When to use

- Eval conclusions flip between runs.
- Compared runs have different seeds or no recorded seed.
- An agent is choosing between prompts, models, or configs based on evals.

## When not to use

- The tests are deterministic unit tests (fix isolation, not seeding).
- The nondeterminism comes from external services or timing, not RNG (pinning the seed will not help; control the environment instead).
- You are doing exploratory single runs to smoke-test a setup (no comparison being made).

## Variant phrasings

### Agent cherry-picked the best of five unseeded runs

Same fix: matched seeds plus a rule against reporting max-of-N as the headline number.

### Eval harness defaults to time-based seeds

Time-based seeding guarantees mismatched comparisons. Override the default in the harness config.

### Nonzero temperature makes "same seed" still nondeterministic

For LLM-output evals, pin the seed AND set temperature to 0 for comparisons, or budget enough trials to average out sampling noise.

## Why it happens

Eval harnesses often seed from the clock by default, and agents rarely check. Each run is then a different draw, and with noisy tasks the winner is whoever got lucky. The agent reports the number in front of it, and nobody notices the comparison was never valid.

## Edge cases

- Some libraries ignore the global seed and use their own RNG; audit every RNG source in the pipeline, not just the top-level seed.
- Parallel workers can consume RNG state in different orders; fix worker count as well as seed.
- GPU nondeterminism can survive seed pinning; for exact reproducibility, also pin the deterministic-ops flags the framework offers.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qqzMULA_fbqQwWCIQjwrfw