Gate deploys on evals so no prompt or model change regresses silently

TL;DR

Run with-arm and without-arm. Set a threshold and a cost cap.

Steps

1. Make evals a build gate, not a dashboard nobody reads.
2. Pin the models. Record the exact model id under test and the judge model id in the eval config. Unpinned model aliases can shift under you and turn a green suite red overnight with no code change.
3. Run with-arm and without-arm. For agent or plugin behavior, run each case twice: once with the new prompt/plugin loaded and once without. The gap between the two is the real signal; a high absolute score with a near-zero gap means your change did nothing.
4. Set a threshold and a cost cap. Fail the build below a score threshold (for example 0.8) and above a dollar cap per run (for example 20). Eval runs and judge calls are real billed usage, so the cap keeps a broken loop from burning money in CI.
5. Keep graders cheap where possible. Regex, required-tool-called, tool-order, and file-exists graders cost nothing because they read the transcript and files on disk. Reserve the LLM judge and baseline-comparison graders for what only a model can check.
6. Run on every prompt or model change. Trigger the suite on pull requests that touch prompts, system instructions, tool definitions, or the model id. A model version bump is a code change for eval purposes.
7. Investigate a near-zero delta before raising the threshold. If scores are flat and the tool-used grader fails, the agent is not choosing the tool on natural phrasing: fix the tool description or the prompt, do not lower the bar.

When to use

You are seeing this: Make evals a build gate, not a dashboard nobody reads. Use this skill when you run into "Gate deploys on evals so no prompt or model change regresses silently".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Claims verified against Anthropic's official docs (platform.claude.com) on 2026-09-26. Evidence URLs in public_references.