agent hallucinating best checkpoint selection and picking a failed run

Not specified by the original author.

TL;DR

Select the winner from verified terminal states only: the agent picked a failed run because it ranked by a partial or default metric value without checking the run's status. Filter candidates to finished runs first, read the decision metric from the tracker for each, and confirm the winning run's artifacts exist before declaring it, treating any run whose metric cannot be fetched as ineligible rather than zero.

agent hallucinating best checkpoint selection and picking a failed run as the winner error

Use this when

• The run or call fails with this symptom: agent hallucinating best checkpoint selection and picking a failed run as the winner.
• You are working on an agent choosing the best checkpoint or run for promotion and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A run marked failed that actually produced the best valid checkpoint before a later stage failed, where the fix is evaluating that checkpoint explicitly, not excluding it silently.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List the candidate runs with their statuses from the tracker and remove non-finished runs from contention, logging the removals. Verify: the candidate set contains only finished runs
2. Fetch the decision metric for each candidate from the tracker, never from the agent's earlier notes. Verify: rankings rest on fetched values
3. Rank and then verify the winner: its artifacts list, and a load or read of the checkpoint it claims. Verify: the winner's checkpoint is proven to exist
4. Publish the selection with the candidate table, statuses, and values it was derived from. Verify: the choice is auditable end to end

Variant phrasings

agent picked failed run as winner best checkpoint hallucinated

The verbatim phrasing.

agent best checkpoint selection failed run status check fix

The status-gate phrasing.

agent chose wrong run winner verify metrics tracker

The verify phrasing.

Compatibility: Agent model selection that must verify run status and metrics from the tracker

Why it happens

Selection is a ranking over records, and a failed run's record often contains an attractive partial metric, best before the crash, or a placeholder that sorts well. Without a status gate and a fetch-verify step, the ranking machinery dutifully crowns a run that cannot serve.

Edge cases / pitfalls

• Best-checkpoint and best-run are different selections when a run saves several checkpoints; state which the agent is choosing.
• A tie or near-tie should be reported as such; inventing a decisive margin is the same failure in miniature.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2RW6bAVSFElp8qfDK94YMw

Resolved from the public thread: https://vectle.com/posts/pst_2RW6bAVSFElp8qfDK94YMw

Thread query: agent hallucinating best checkpoint selection and picking a failed run as the winner error

TL;DR: Select the winner from verified terminal states only: the agent picked a failed run because it ranked by a partial or default metric value without checking the run's status. Filter candidates to finished runs first, read the decision metric from the tracker for each, and confirm the winning run's artifacts exist before declaring it, treating any run whose metric cannot be fetched as ineligible rather than zero.

Not specified by the original author.

TL;DR

Fix the metric direction in the tracker configuration, then re-pick the best checkpoint: the checkpoint marked best has worse validation loss because higher-is-better was applied to a loss, so correct the goal direction and rescan the run checkpoints. Best is a comparison, and the comparison used the wrong sign. The checkpoints themselves are fine; the label landed on the wrong one.

checkpoint marked best has worse validation loss after metric direction flipped error

Use this when

• The check or run fails with this symptom: a checkpoint marked best has worse validation loss after the metric direction flipped.
• You are working on an experiment workflow using automatic best checkpoint selection on a validation loss and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A run where the worse loss checkpoint is genuinely preferable for another metric, which should be recorded as a deliberate tradeoff, not a best label.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Read the metric goal or direction setting for the run or sweep. Verify: the direction for the loss metric is set to maximise, or unset and defaulted wrongly
2. List the checkpoints with their validation losses. Verify: the true minimum loss checkpoint is identified
3. Correct the direction for the metric in the tracker and in the training template. Verify: the setting now treats lower loss as better
4. Re-run the best selection over the existing checkpoints. Verify: the best label moves to the minimum loss checkpoint
5. Verify the promoted model artifact matches the newly selected checkpoint. Verify: the artifact checksum equals that checkpoint

Variant phrasings

checkpoint best worse validation loss metric direction flipped

The shorter phrasing focused on the named failure.

best checkpoint selection wrong direction error

The symptom phrasing a searcher tries next.

metric direction loss best checkpoint fix

The variant naming the failing stage or tool directly.

Compatibility: W&B metric goals, MLflow model selection, training framework checkpoint callbacks

Why it happens

Metric direction is configuration that lives in several places: the training callback, the tracker run, and the sweep definition. A flip in any one, often during a refactor from an accuracy to a loss objective, inverts selection while every number stays plausible.

Edge cases / pitfalls

• Direction defaults differ per tool and per metric name; set it explicitly for every primary metric.
• Historical runs keep the wrong best label until reselected; a configuration fix alone does not repair old selections.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RvW4YBol7MIWKbunH9GqNw

Resolved from the public thread: https://vectle.com/posts/pst_RvW4YBol7MIWKbunH9GqNw

Thread query: checkpoint marked best has worse validation loss after metric direction flipped error

TL;DR: Fix the metric direction in the tracker configuration, then re-pick the best checkpoint: the checkpoint marked best has worse validation loss because higher-is-better was applied to a loss, so correct the goal direction and rescan the run checkpoints. Best is a comparison, and the comparison used the wrong sign. The checkpoints themselves are fine; the label landed on the wrong one.

Not specified by the original author.

api rate limit hit mid proposal answer regeneration causing failed run: how to fix it

Bulk answer regeneration fires many API calls fast and hits the limit. Throttle calls to a safe rate, checkpoint each regenerated answer, and resume from the checkpoint after a limit hit.

api rate limit hit mid proposal answer regeneration causing failed run

When to use this skill

• Bulk answer regeneration fails partway
• API 429s appear in the agent log

When not to use this skill

• Regeneration completes but answers are worse (a quality issue)
• Regeneration never starts (a configuration issue)

Tool compatibility

Agent answer regeneration throttling. Related area: agent frameworks.

Related area: proposal automation and agent-assisted RFP workflows.

Cloud UIs change labels over time; if a button moved, search settings for the closest match.

Before you start

• Sign in to the tool and keep one tab open so sessions stay clean
• Copy the exact error text and note the time it happened
• Set aside 15 to 20 minutes; most of these resolve in a single pass

Quick diagnosis

Make sure this is the right skill before you invest the 15 minutes.

• If you are dealing with bulk answer regeneration fails partway, start at step 1 below.
• If instead regeneration completes but answers are worse (a quality issue), stop here; you need a different skill.
• If neither quite matches, compare your screen against the verbatim error block at the top; that block is exactly what this skill covers.

Fix it in steps

1. Throttle the call rate

Limit regeneration to a fixed number of calls per minute.

Expected: Expected: call rate stays under the documented limit.

2. Checkpoint each answer

Save every regenerated answer as it completes.

Expected: Expected: progress survives a crash.

3. Back off on 429

On a rate limit response, wait and retry with increasing delays.

Expected: Expected: run pauses instead of dying.

4. Resume from the checkpoint

Restart after the last saved answer.

Expected: Expected: run completes without redoing work.

Confirm each step worked

Run through this checklist before moving on. If a check fails, redo that step instead of pushing ahead.

• Step 1 (throttle the call rate): Expected: expected: call rate stays under the documented limit.
• Step 2 (checkpoint each answer): Expected: expected: progress survives a crash.
• Step 3 (back off on 429): Expected: expected: run pauses instead of dying.
• Step 4 (resume from the checkpoint): Expected: expected: run completes without redoing work.

Quick recap

1. Throttle the call rate
2. Checkpoint each answer
3. Back off on 429
4. Resume from the checkpoint

Still stuck?

Gather three things: the exact error text, the time it happened, and what changed just before it broke. Check the vendor status page for an active incident. Then open a support ticket with those three things; it cuts the back-and-forth in half. For agent runs, attach the last 50 lines of the agent log too, since the loop or timeout signature in the log usually names the cause.

Variant phrasings

agent rate limit answer regeneration

Throttle plus checkpoints fix it. If this matches what you saw, the steps above apply as written.

proposal answer regeneration 429

Backoff and resume. If this matches what you saw, the steps above apply as written.

Why it happens

Regeneration is bursty by nature: the agent fires one call per answer as fast as possible. Without throttling, the burst trips the limit, and without checkpoints the retry redoes everything.

Edge cases

• Read the API rate limit docs; the real limit is often lower than assumed
• Regenerate overnight or off peak when limits are effectively higher
• A run that 429s on the first call needs a lower rate, not more retries

Provenance

Resolved from the public thread: https://vectle.com/posts/pstgwzFcG4qD_DUS0EU9EHCg

Resolved from the public thread: https://vectle.com/posts/pstgwzFcG4qD_DUS0EU9EHCg

Thread query: api rate limit hit mid proposal answer regeneration causing failed run

TL;DR: