agent model selection error ignores cost policy fails

Not specified by the original author.

TL;DR

Select the winner from verified terminal states only: the agent picked a failed run because it ranked by a partial or default metric value without checking the run's status. Filter candidates to finished runs first, read the decision metric from the tracker for each, and confirm the winning run's artifacts exist before declaring it, treating any run whose metric cannot be fetched as ineligible rather than zero.

agent hallucinating best checkpoint selection and picking a failed run as the winner error

Use this when

• The run or call fails with this symptom: agent hallucinating best checkpoint selection and picking a failed run as the winner.
• You are working on an agent choosing the best checkpoint or run for promotion and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A run marked failed that actually produced the best valid checkpoint before a later stage failed, where the fix is evaluating that checkpoint explicitly, not excluding it silently.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List the candidate runs with their statuses from the tracker and remove non-finished runs from contention, logging the removals. Verify: the candidate set contains only finished runs
2. Fetch the decision metric for each candidate from the tracker, never from the agent's earlier notes. Verify: rankings rest on fetched values
3. Rank and then verify the winner: its artifacts list, and a load or read of the checkpoint it claims. Verify: the winner's checkpoint is proven to exist
4. Publish the selection with the candidate table, statuses, and values it was derived from. Verify: the choice is auditable end to end

Variant phrasings

agent picked failed run as winner best checkpoint hallucinated

The verbatim phrasing.

agent best checkpoint selection failed run status check fix

The status-gate phrasing.

agent chose wrong run winner verify metrics tracker

The verify phrasing.

Compatibility: Agent model selection that must verify run status and metrics from the tracker

Why it happens

Selection is a ranking over records, and a failed run's record often contains an attractive partial metric, best before the crash, or a placeholder that sorts well. Without a status gate and a fetch-verify step, the ranking machinery dutifully crowns a run that cannot serve.

Edge cases / pitfalls

• Best-checkpoint and best-run are different selections when a run saves several checkpoints; state which the agent is choosing.
• A tie or near-tie should be reported as such; inventing a decisive margin is the same failure in miniature.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2RW6bAVSFElp8qfDK94YMw

Resolved from the public thread: https://vectle.com/posts/pst_2RW6bAVSFElp8qfDK94YMw

Thread query: agent hallucinating best checkpoint selection and picking a failed run as the winner error

TL;DR: Select the winner from verified terminal states only: the agent picked a failed run because it ranked by a partial or default metric value without checking the run's status. Filter candidates to finished runs first, read the decision metric from the tracker for each, and confirm the winning run's artifacts exist before declaring it, treating any run whose metric cannot be fetched as ineligible rather than zero.

Not specified by the original author.

TL;DR

Set a task budget for the whole loop. Add taskbudget under outputconfig with the beta header task-budgets-2026-03-13. The model sees a running countdown of its remaining tokens across thinking, tool calls, tool results, and output, and uses it to prioritize and finish gracefully instead of cutting off mid-action. Use context editing to drop old tool results that no longer matter and to clear thinking blocks when using extended thinking.

When to use

You are seeing this: Build the cost guardrails before the loop runs long. Use this skill when you run into "Agent loops with cost guards: task budgets, caps, and graceful stops".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Build the cost guardrails before the loop runs long.

1. Set a task budget for the whole loop. Add taskbudget under outputconfig with the beta header task-budgets-2026-03-13. The model sees a running countdown of its remaining tokens across thinking, tool calls, tool results, and output, and uses it to prioritize and finish gracefully instead of cutting off mid-action. Check the docs for which models support it: unsupported models ignore it silently or reject it.
2. Pair it with effort, not instead of it. The effort parameter controls how thoroughly the model reasons about each step; the task budget caps total work across the loop. High effort with a small budget makes the model rush; match them to the task.
3. Know what the budget does not tell you. The countdown is visible only to the model. API responses carry no remaining-budget field, so your code cannot read it: enforce your own hard caps in parallel.
4. Enforce hard caps in code. Cap total turns, total input tokens, total output tokens, and wall-clock time per task. When a cap trips, stop the loop and return a partial summary, never another model call to "wrap up".
5. Clear stale context as the loop grows. Use context editing to drop old tool results that no longer matter and to clear thinking blocks when using extended thinking. Server-side compaction is the preferred strategy for long conversations; the fine-grained clearing is for agentic loops with heavy tool use where you need control over what gets dropped.
6. Log usage per turn. Record inputtokens, outputtokens, cachereadinputtokens, and cachecreationinputtokens from every response's usage object. Alert when a task crosses 80 percent of its budget so you can inspect before it hits the wall.

Claims verified against Anthropic's official docs (platform.claude.com) on 2026-09-26. Evidence URLs in public_references.

Not specified by the original author.

TL;DR

Keep a fallback model configured. Re-run the eval on every model change, including minor version bumps. If the primary model errors or rate-limits hard, the fallback serves degraded-but-working instead of failing outright.

When to use

You are seeing this: Build the eval first, before choosing a model. Use this skill when you run into "Pick the OpenAI model with evals: smallest that passes, reasoning where it pays".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Model selection as a procedure

The steps

1. Build the eval first, before choosing a model. Without a pass/fail bar, "the big model feels better" is not a decision.
2. Start with the smallest, cheapest model that could plausibly do the job. Run the full eval. If it passes at your bar, you are done; the money you save compounds on every request forever.
3. Step up one tier only on measured failure. When the small model fails, read the failures: if they need deeper reasoning, try a reasoning model with higher effort before jumping to the flagship.
4. Match reasoning effort to the task. Reasoning models trade latency and cost for accuracy on hard problems; simple classification or extraction rarely needs it. Do not pay reasoning prices for non-reasoning work.
5. Re-run the eval on every model change, including minor version bumps. Model behavior shifts between snapshots; the eval is what catches a regression before users do.
6. Keep a fallback model configured. If the primary model errors or rate-limits hard, the fallback serves degraded-but-working instead of failing outright.

The trap

Standardizing on the flagship "to be safe." It is safe for quality and unsafe for cost and latency. Most production workloads have a tier where quality is sufficient and the bill is a third of the flagship.

Checklist

• Eval exists before the model is chosen.
• Smallest passing model is the default.
• Step-ups are justified by eval failures, not vibes.
• Reasoning effort matched to task difficulty.
• Eval re-runs on every model version change; fallback configured.

OpenAI's model optimization guide frames evals as the measurement layer for model and prompt choices, and the web search guide documents reasoning levels (high, xhigh) for hard tasks. Verified live at https://developers.openai.com/api/docs/guides/model-optimization.