Gate deploys on evals so no prompt or model change regresses silently

Make evals a build gate, not a dashboard nobody reads.

1. Pin the models. Record the exact model id under test and the judge model id in the eval config. Unpinned model aliases can shift under you and turn a green suite red overnight with no code change.
2. Run with-arm and without-arm. For agent or plugin behavior, run each case twice: once with the new prompt/plugin loaded and once without. The gap between the two is the real signal; a high absolute score with a near-zero gap means your change did nothing.
3. Set a threshold and a cost cap. Fail the build below a score threshold (for example 0.8) and above a dollar cap per run (for example 20). Eval runs and judge calls are real billed usage, so the cap keeps a broken loop from burning money in CI.
4. Keep graders cheap where possible. Regex, required-tool-called, tool-order, and file-exists graders cost nothing because they read the transcript and files on disk. Reserve the LLM judge and baseline-comparison graders for what only a model can check.
5. Run on every prompt or model change. Trigger the suite on pull requests that touch prompts, system instructions, tool definitions, or the model id. A model version bump is a code change for eval purposes.
6. Investigate a near-zero delta before raising the threshold. If scores are flat and the tool-used grader fails, the agent is not choosing the tool on natural phrasing: fix the tool description or the prompt, do not lower the bar.

Claims verified against Anthropic's official docs (platform.claude.com) on 2026-09-26. Evidence URLs in public_references.