Define success criteria before you tune a single Claude prompt

TL;DR

Use code where you can: exact match, regex, required tool called, required file exists. Most use cases need evaluation along several success criteria: correctness, citation accuracy, tone, latency, cost per task. Use an LLM judge only where code cannot reach (tone, nuance, open-ended quality). When you use a judge, recalibrate it every time you change the model under test, judges drift with the model.

When to use

You are seeing this: Do this before writing or changing any prompt. Use this skill when you run into "Define success criteria before you tune a single Claude prompt".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Do this before writing or changing any prompt.

1. Write success criteria as sentences you can measure. Good criteria are specific, measurable, achievable, and relevant. "Good performance" is not a criterion. "The refund answer cites the correct policy section in 95 percent of cases" is.
2. Go multidimensional. Most use cases need evaluation along several success criteria: correctness, citation accuracy, tone, latency, cost per task. One number hides regressions in the others.
3. Build cases from real failures, bottom-up. Collect 20 to 50 real inputs where the current behavior was wrong or borderline, including edge cases and prompt-injection attempts if the prompt handles external content. Design evals that mirror your real-world task distribution, not clean textbook examples.
4. Choose graders per criterion. Use code where you can: exact match, regex, required tool called, required file exists. Use an LLM judge only where code cannot reach (tone, nuance, open-ended quality). When you use a judge, recalibrate it every time you change the model under test, judges drift with the model.
5. Prefer volume over perfect labels. Automate when possible and accept noisy labels at volume over a handful of hand-perfect cases. A 200-case automated suite with 90 percent label accuracy beats a 10-case perfect one for catching regressions.
6. Freeze the set before tuning. Lock the cases and graders, record a baseline score, then start changing prompts. Any change to the eval set after tuning starts must be logged as a new version, or you will tune to a moving target.

Claims verified against Anthropic's official docs (platform.claude.com) on 2026-09-26. Evidence URLs in public_references.