Skip to content
← Back to activity

Immutable solution version

OpenAI evals: dataset, graders, run, iterate, the loop that makes models shippable

Version 1Sign in to save

Complete solution

An eval is not a one-time report. It is a loop: representative dataset, graders that check what matters, run against each model change, and fix what fails. Build the loop before you need it.

The eval loop

The steps

  1. Collect a dataset of real inputs with known-good outputs. Pull from production logs, support tickets, or wherever your actual traffic lives. Synthetic examples you wrote yourself will flatter the model.
  2. Start with a string_check grader for anything with an exact answer: labels, categories, extracted fields. Exact-match grading is cheap and unambiguous; only reach for model graders when the output is genuinely open-ended.
  3. Define testing_criteria per item: what counts as correct for this specific case. One global rubric for a diverse dataset grades nothing well.
  4. Run the eval against your current model and prompt. Read the failures, not the score. A 90% pass rate tells you nothing; the ten failures tell you what to fix.
  5. Fix the prompt, the retrieval, or the data behind each failure class, then re-run. The eval is the gate: no model swap, prompt change, or fine-tune ships without passing it.
  6. Version the dataset alongside the prompt. When the eval and the code drift apart, the eval stops meaning anything.

The trap

Running evals once, celebrating the score, and never running them again. Models change, prompts change, data drifts. An eval that does not run on every change is decoration.

Checklist

  • Dataset items come from real traffic.
  • Graders start with string_check where answers are exact.
  • testing_criteria is per-item, not one global rubric.
  • The eval runs on every model or prompt change.
  • Dataset and prompt are versioned together.

Evidence

OpenAI's evals guide documents datasets with items, testingcriteria, and the stringcheck grader. Verified live at https://developers.openai.com/api/docs/guides/evals.

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_VEibOrD8KZZQSfmKZKicig" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">OpenAI evals: dataset, graders, run, iterate, the loop that makes models shippable <span>— Vectle</span></a>