OpenAI evals: dataset, graders, run, iterate, the loop that makes models shippable

TL;DR

Run the eval against your current model and prompt. Collect a dataset of real inputs with known-good outputs.

Steps

1. Collect a dataset of real inputs with known-good outputs. Pull from production logs, support tickets, or wherever your actual traffic lives. Synthetic examples you wrote yourself will flatter the model.
2. Start with a string_check grader for anything with an exact answer: labels, categories, extracted fields. Exact-match grading is cheap and unambiguous; only reach for model graders when the output is genuinely open-ended.
3. Define testing_criteria per item: what counts as correct for this specific case. One global rubric for a diverse dataset grades nothing well.
4. Run the eval against your current model and prompt. Read the failures, not the score. A 90% pass rate tells you nothing; the ten failures tell you what to fix.
5. Fix the prompt, the retrieval, or the data behind each failure class, then re-run. The eval is the gate: no model swap, prompt change, or fine-tune ships without passing it.
6. Version the dataset alongside the prompt. When the eval and the code drift apart, the eval stops meaning anything.
7. Running evals once, celebrating the score, and never running them again. Models change, prompts change, data drifts. An eval that does not run on every change is decoration.
8. - Dataset items come from real traffic.

• Graders start with string_check where answers are exact.
• testing_criteria is per-item, not one global rubric.
• The eval runs on every model or prompt change.
• Dataset and prompt are versioned together.

When to use

You are seeing this: Pull from production logs, support tickets, or wherever your actual traffic lives. Use this skill when you run into "OpenAI evals: dataset, graders, run, iterate, the loop that makes models shippable".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

OpenAI's evals guide documents datasets with items, testingcriteria, and the stringcheck grader. Verified live at https://developers.openai.com/api/docs/guides/evals.