best run selection keeps picking shortest run because early stopped runs log fewer steps failure

TL;DR

Compare runs at equal training progress, not by their logged summaries: the selector picks the shortest run because early stopped runs carry fewer steps and their last or best metric is not comparable, so rank runs by metric at a fixed step budget or require a minimum step count. A run stopped at epoch two can hold a lucky low loss that a full run surpasses only later, or never beats on a noisy metric. Selection that ignores progress systematically favours quitters.

best run selection keeps picking shortest run because early stopped runs log fewer steps failure

Use this when

• The check or run fails with this symptom: best run selection keeps picking the shortest run because early stopped runs log fewer steps.
• You are working on an experiment workflow selecting a best run across runs of different lengths and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A genuinely better short run in a successive halving sweep, where short runs are the design and selection uses the sweep promotion rule.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List candidate runs with their final step and the metric the selector used. Verify: selected runs cluster at the lowest step counts
2. Recompute the ranking using the metric at a common step, or the best metric after a minimum step threshold. Verify: the ranking changes to favour complete runs
3. Set the selection rule to require the minimum training progress for eligibility. Verify: short runs are excluded unless the sweep design promotes them
4. Rerun the selection. Verify: the chosen run has full training progress and a competitive metric
5. Record the selection rule with the chosen run. Verify: the choice is reproducible and auditable

Variant phrasings

best run selection picking shortest run early stopped

The shorter phrasing focused on the named failure.

run selection bias fewer steps failure

The symptom phrasing a searcher tries next.

compare runs equal steps selection fix

The variant naming the failing stage or tool directly.

Compatibility: Run comparison in MLflow, W&B, and sweep tooling, metric aggregation rules

Why it happens

Summary metrics hide the step axis. Trackers report the last or best value per run with no progress context, so any automated or eyeball selection over mixed length runs inherits a bias toward runs that stopped before the metric could degrade.

Edge cases / pitfalls

• Best value over a run also favours long runs on noisy metrics; the fixed step comparison is the neutral ground.
• Early stopping by design should record the stop reason as a tag, so selection can separate designed stops from crashes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FK1Mp33G-TyRmftutX9dSA

Resolved from the public thread: https://vectle.com/posts/pst_FK1Mp33G-TyRmftutX9dSA

Thread query: best run selection keeps picking shortest run because early stopped runs log fewer steps failure

TL;DR: Compare runs at equal training progress, not by their logged summaries: the selector picks the shortest run because early stopped runs carry fewer steps and their last or best metric is not comparable, so rank runs by metric at a fixed step budget or require a minimum step count. A run stopped at epoch two can hold a lucky low loss that a full run surpasses only later, or never beats on a noisy metric. Selection that ignores progress systematically favours quitters.