agent hallucinating best checkpoint selection and picking a failed run as the winner error

Not specified by the original author.

TL;DR

Select the winner from verified terminal states only: the agent picked a failed run because it ranked by a partial or default metric value without checking the run's status. Filter candidates to finished runs first, read the decision metric from the tracker for each, and confirm the winning run's artifacts exist before declaring it, treating any run whose metric cannot be fetched as ineligible rather than zero.

agent hallucinating best checkpoint selection and picking a failed run as the winner error

Use this when

• The run or call fails with this symptom: agent hallucinating best checkpoint selection and picking a failed run as the winner.
• You are working on an agent choosing the best checkpoint or run for promotion and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A run marked failed that actually produced the best valid checkpoint before a later stage failed, where the fix is evaluating that checkpoint explicitly, not excluding it silently.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List the candidate runs with their statuses from the tracker and remove non-finished runs from contention, logging the removals. Verify: the candidate set contains only finished runs
2. Fetch the decision metric for each candidate from the tracker, never from the agent's earlier notes. Verify: rankings rest on fetched values
3. Rank and then verify the winner: its artifacts list, and a load or read of the checkpoint it claims. Verify: the winner's checkpoint is proven to exist
4. Publish the selection with the candidate table, statuses, and values it was derived from. Verify: the choice is auditable end to end

Variant phrasings

agent picked failed run as winner best checkpoint hallucinated

The verbatim phrasing.

agent best checkpoint selection failed run status check fix

The status-gate phrasing.

agent chose wrong run winner verify metrics tracker

The verify phrasing.

Compatibility: Agent model selection that must verify run status and metrics from the tracker

Why it happens

Selection is a ranking over records, and a failed run's record often contains an attractive partial metric, best before the crash, or a placeholder that sorts well. Without a status gate and a fetch-verify step, the ranking machinery dutifully crowns a run that cannot serve.

Edge cases / pitfalls

• Best-checkpoint and best-run are different selections when a run saves several checkpoints; state which the agent is choosing.
• A tie or near-tie should be reported as such; inventing a decisive margin is the same failure in miniature.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2RW6bAVSFElp8qfDK94YMw

Resolved from the public thread: https://vectle.com/posts/pst_2RW6bAVSFElp8qfDK94YMw

Thread query: agent hallucinating best checkpoint selection and picking a failed run as the winner error

TL;DR: Select the winner from verified terminal states only: the agent picked a failed run because it ranked by a partial or default metric value without checking the run's status. Filter candidates to finished runs first, read the decision metric from the tracker for each, and confirm the winning run's artifacts exist before declaring it, treating any run whose metric cannot be fetched as ineligible rather than zero.