# The eval loop
## The steps
1. Collect a dataset of real inputs with known-good outputs. Pull from production logs, support tickets, or wherever your actual traffic lives. Synthetic examples you wrote yourself will flatter the model.
2. Start with a `string_check` grader for anything with an exact answer: labels, categories, extracted fields. Exact-match grading is cheap and unambiguous; only reach for model graders when the output is genuinely open-ended.
3. Define `testing_criteria` per item: what counts as correct for this specific case. One global rubric for a diverse dataset grades nothing well.
4. Run the eval against your current model and prompt. Read the failures, not the score. A 90% pass rate tells you nothing; the ten failures tell you what to fix.
5. Fix the prompt, the retrieval, or the data behind each failure class, then re-run. The eval is the gate: no model swap, prompt change, or fine-tune ships without passing it.
6. Version the dataset alongside the prompt. When the eval and the code drift apart, the eval stops meaning anything.
## The trap
Running evals once, celebrating the score, and never running them again. Models change, prompts change, data drifts. An eval that does not run on every change is decoration.
## Checklist
- Dataset items come from real traffic.
- Graders start with string_check where answers are exact.
- testing_criteria is per-item, not one global rubric.
- The eval runs on every model or prompt change.
- Dataset and prompt are versioned together.