Do this before writing or changing any prompt.
1. Write success criteria as sentences you can measure. Good criteria are specific, measurable, achievable, and relevant. "Good performance" is not a criterion. "The refund answer cites the correct policy section in 95 percent of cases" is.
2. Go multidimensional. Most use cases need evaluation along several success criteria: correctness, citation accuracy, tone, latency, cost per task. One number hides regressions in the others.
3. Build cases from real failures, bottom-up. Collect 20 to 50 real inputs where the current behavior was wrong or borderline, including edge cases and prompt-injection attempts if the prompt handles external content. Design evals that mirror your real-world task distribution, not clean textbook examples.
4. Choose graders per criterion. Use code where you can: exact match, regex, required tool called, required file exists. Use an LLM judge only where code cannot reach (tone, nuance, open-ended quality). When you use a judge, recalibrate it every time you change the model under test, judges drift with the model.
5. Prefer volume over perfect labels. Automate when possible and accept noisy labels at volume over a handful of hand-perfect cases. A 200-case automated suite with 90 percent label accuracy beats a 10-case perfect one for catching regressions.
6. Freeze the set before tuning. Lock the cases and graders, record a baseline score, then start changing prompts. Any change to the eval set after tuning starts must be logged as a new version, or you will tune to a moving target.