LLM eval harness run fails with dataset not found error scoring broken

Not specified by the original author.

TL;DR

Run with-arm and without-arm. Set a threshold and a cost cap.

Steps

1. Make evals a build gate, not a dashboard nobody reads.
2. Pin the models. Record the exact model id under test and the judge model id in the eval config. Unpinned model aliases can shift under you and turn a green suite red overnight with no code change.
3. Run with-arm and without-arm. For agent or plugin behavior, run each case twice: once with the new prompt/plugin loaded and once without. The gap between the two is the real signal; a high absolute score with a near-zero gap means your change did nothing.
4. Set a threshold and a cost cap. Fail the build below a score threshold (for example 0.8) and above a dollar cap per run (for example 20). Eval runs and judge calls are real billed usage, so the cap keeps a broken loop from burning money in CI.
5. Keep graders cheap where possible. Regex, required-tool-called, tool-order, and file-exists graders cost nothing because they read the transcript and files on disk. Reserve the LLM judge and baseline-comparison graders for what only a model can check.
6. Run on every prompt or model change. Trigger the suite on pull requests that touch prompts, system instructions, tool definitions, or the model id. A model version bump is a code change for eval purposes.
7. Investigate a near-zero delta before raising the threshold. If scores are flat and the tool-used grader fails, the agent is not choosing the tool on natural phrasing: fix the tool description or the prompt, do not lower the bar.

When to use

You are seeing this: Make evals a build gate, not a dashboard nobody reads. Use this skill when you run into "Gate deploys on evals so no prompt or model change regresses silently".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Claims verified against Anthropic's official docs (platform.claude.com) on 2026-09-26. Evidence URLs in public_references.

Not specified by the original author.

TL;DR

Run the eval against your current model and prompt. Collect a dataset of real inputs with known-good outputs.

Steps

1. Collect a dataset of real inputs with known-good outputs. Pull from production logs, support tickets, or wherever your actual traffic lives. Synthetic examples you wrote yourself will flatter the model.
2. Start with a string_check grader for anything with an exact answer: labels, categories, extracted fields. Exact-match grading is cheap and unambiguous; only reach for model graders when the output is genuinely open-ended.
3. Define testing_criteria per item: what counts as correct for this specific case. One global rubric for a diverse dataset grades nothing well.
4. Run the eval against your current model and prompt. Read the failures, not the score. A 90% pass rate tells you nothing; the ten failures tell you what to fix.
5. Fix the prompt, the retrieval, or the data behind each failure class, then re-run. The eval is the gate: no model swap, prompt change, or fine-tune ships without passing it.
6. Version the dataset alongside the prompt. When the eval and the code drift apart, the eval stops meaning anything.
7. Running evals once, celebrating the score, and never running them again. Models change, prompts change, data drifts. An eval that does not run on every change is decoration.
8. - Dataset items come from real traffic.

• Graders start with string_check where answers are exact.
• testing_criteria is per-item, not one global rubric.
• The eval runs on every model or prompt change.
• Dataset and prompt are versioned together.

When to use

You are seeing this: Pull from production logs, support tickets, or wherever your actual traffic lives. Use this skill when you run into "OpenAI evals: dataset, graders, run, iterate, the loop that makes models shippable".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

OpenAI's evals guide documents datasets with items, testingcriteria, and the stringcheck grader. Verified live at https://developers.openai.com/api/docs/guides/evals.

Not specified by the original author.

TL;DR

If it is missing facts, build the minimal RAG: upload the docs, wire file_search, and run your eval. If it is behavior, try the prompt first: rewrite the developer instructions with concrete examples of good and bad output, run the eval.

Steps

1. Write down what is actually failing: the model lacks facts it could look up, or the model knows the facts but behaves wrong (tone, format, consistency, refusal style).
2. If it is missing facts, build the minimal RAG: upload the docs, wire file_search, and run your eval. Measure before assuming fine-tuning would do better.
3. If it is behavior, try the prompt first: rewrite the developer instructions with concrete examples of good and bad output, run the eval. Most behavior problems die here.
4. Only if the prompt cannot hold the behavior consistently across your eval set, fine-tune on examples of the desired behavior. Fine-tuning is for what prompting cannot reliably produce.
5. Never fine-tune to teach stable facts that change. A fine-tuned model bakes in the training snapshot; docs that update monthly need retrieval, not weights.
6. Fine-tuning as the first resort. It is the slowest, most expensive fix, and it cannot be patched quickly when the world changes. The order is always: prompt, then retrieval, then fine-tune, with an eval gating each step.
7. - The failure is classified as knowledge vs behavior before any work starts.

• RAG was tried for knowledge gaps, prompt rewrite for behavior gaps.
• Fine-tuning only after the cheaper fixes demonstrably failed on the eval.
• Nothing that changes monthly is baked into weights.

When to use

You are seeing this: Write down what is actually failing: the model lacks facts it could look up, or the model knows the facts but behaves wrong (tone, format, consistency, refusal style). Use this skill when you run into "Decide RAG vs fine-tuning with an eval, not a hunch".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

OpenAI's model optimization guide frames the workflow as evals first, then prompt engineering, then fine-tuning. Verified live at https://developers.openai.com/api/docs/guides/model-optimization and https://developers.openai.com/api/docs/guides/supervised-fine-tuning.