agent's braintrust eval dataset changed mid-run: version pinning

TL;DR

If the eval dataset changes mid-run, every comparison from that run is garbage. Pin the dataset version (or content hash) when the run starts, verify it before scoring, and fail the run if it changed. Dataset evolution happens through versioned releases with a re-baseline, never through silent mutation.

Error

agent's braintrust eval dataset changed mid-run: version pinning

Scores swung between runs because rows were added to the dataset while the eval was in flight; the agent reported the numbers as comparable.

Steps

1. Capture the dataset identity at run start: version tag plus a content hash of the rows. Expected output: a pinned dataset fingerprint in the run metadata.
2. Re-verify the fingerprint before scoring and before comparing against other runs. Expected output: a check that passes silently or fails the run loudly.
3. If the fingerprint changed, invalidate the run: do not report scores, do not compare, mark it contaminated. Expected output: a failed run with the cause named, not a misleading report.
4. Move dataset edits to versioned releases: every change mints a new version, and comparisons only happen within a version. Expected output: a dataset changelog with versions.
5. Re-baseline on version bumps: when the dataset version changes deliberately, re-run the baseline so old and new scores are never mixed. Expected output: per-version baselines, no cross-version comparisons.

When to use

• Eval scores shift without any code or prompt change.
• Multiple people or jobs write to the dataset the eval reads.
• An agent's eval report compares runs from different days.

When not to use

• The dataset is intentionally append-only with timestamped rows and the eval filters by time window (then the window is the pin; still record it).
• You are iterating on the dataset itself (then it is dataset development, not an eval run; do not report scores).
• The eval is a one-off smoke check with no comparisons.

Variant phrasings

Someone edited the dataset between the baseline and the candidate run

The classic invalidation. Same fix: pin at start, verify before compare.

The dataset is a live view, not a snapshot

Live views are unpinnable by nature. Snapshot the view at run start and eval against the snapshot.

Agent compared its run against last week's published scores

Cross-version comparison. Re-run the baseline on the current dataset version first.

Why it happens

Datasets feel static but are often shared, editable, and live. The eval harness reads whatever is there at read time, so a run that starts on version 12 can score on version 13. The agent reports the numbers it got, and nobody checks that the ground moved underneath.

Edge cases

• Content hashing a huge dataset on every run is expensive; hash once at pin time and verify the version tag plus row count cheaply thereafter.
• Concurrent writers can change the dataset between pin and verify; use the platform's snapshot or transaction feature when available.
• Pinned datasets still go stale; schedule periodic re-baselines so the pin does not fossilize.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst9rwdSHoxPIWzYmTVwO5Jg