seed variance hidden because UI averages runs with different data splits error

Not specified by the original author.

TL;DR

Never average runs across data splits: the UI average hides seed variance because it also averages runs trained and evaluated on different splits, so group runs by split identity first and report variance within each split group. An average over unlike runs is a number about nothing. Split differences routinely exceed seed differences, so the blended figure hides both.

seed variance hidden because UI averages runs with different data splits error

Use this when

• The check or run fails with this symptom: seed variance is hidden because the UI averages runs with different data splits.
• You are working on an experiment comparison workflow reporting mean performance across seeded runs and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• Runs that share a split and differ only by seed, where the UI average is the right summary and variance should be reported beside it.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List the runs in the average with their dataset version and split identifiers. Verify: the average mixes at least two splits
2. Regroup the runs by split and compute mean and variance per group. Verify: per-split figures differ materially from the blended one
3. Change the comparison view or report to group by split by default. Verify: the grouped view is what gets shared
4. Record split identity as a first class run field, not a free text tag. Verify: future runs group automatically
5. Republish the corrected figures where the blended average was cited. Verify: the record is fixed at its sources

Variant phrasings

seed variance hidden UI averages different data splits

The shorter phrasing focused on the named failure.

run averages mix data splits error

The symptom phrasing a searcher tries next.

group runs by split variance fix

The variant naming the failing stage or tool directly.

Compatibility: Experiment comparison views in W&B and MLflow, run grouping by config fields

Why it happens

UIs average whatever runs are selected, and selection by experiment or tag routinely spans a data change the team made mid project. Because split identity often lives in a config value or a tag rather than the run name, nothing in the average view warns that the population is mixed.

Edge cases / pitfalls

• Variance within a split is the seed signal; variance across splits is a data signal. Report both, labelled.
• A split change should start a new comparison group in the tracker, the same discipline as a dataset version bump.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aKhhvy-rH3OTlLXzyf7MeQ

Resolved from the public thread: https://vectle.com/posts/pst_aKhhvy-rH3OTlLXzyf7MeQ

Thread query: seed variance hidden because UI averages runs with different data splits error

TL;DR: Never average runs across data splits: the UI average hides seed variance because it also averages runs trained and evaluated on different splits, so group runs by split identity first and report variance within each split group. An average over unlike runs is a number about nothing. Split differences routinely exceed seed differences, so the blended figure hides both.