Pick the OpenAI model with evals: smallest that passes, reasoning where it pays

TL;DR

Keep a fallback model configured. Re-run the eval on every model change, including minor version bumps. If the primary model errors or rate-limits hard, the fallback serves degraded-but-working instead of failing outright.

When to use

You are seeing this: Build the eval first, before choosing a model. Use this skill when you run into "Pick the OpenAI model with evals: smallest that passes, reasoning where it pays".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Model selection as a procedure

The steps

1. Build the eval first, before choosing a model. Without a pass/fail bar, "the big model feels better" is not a decision.
2. Start with the smallest, cheapest model that could plausibly do the job. Run the full eval. If it passes at your bar, you are done; the money you save compounds on every request forever.
3. Step up one tier only on measured failure. When the small model fails, read the failures: if they need deeper reasoning, try a reasoning model with higher effort before jumping to the flagship.
4. Match reasoning effort to the task. Reasoning models trade latency and cost for accuracy on hard problems; simple classification or extraction rarely needs it. Do not pay reasoning prices for non-reasoning work.
5. Re-run the eval on every model change, including minor version bumps. Model behavior shifts between snapshots; the eval is what catches a regression before users do.
6. Keep a fallback model configured. If the primary model errors or rate-limits hard, the fallback serves degraded-but-working instead of failing outright.

The trap

Standardizing on the flagship "to be safe." It is safe for quality and unsafe for cost and latency. Most production workloads have a tier where quality is sufficient and the bill is a third of the flagship.

Checklist

• Eval exists before the model is chosen.
• Smallest passing model is the default.
• Step-ups are justified by eval failures, not vibes.
• Reasoning effort matched to task difficulty.
• Eval re-runs on every model version change; fallback configured.

OpenAI's model optimization guide frames evals as the measurement layer for model and prompt choices, and the web search guide documents reasoning levels (high, xhigh) for hard tasks. Verified live at https://developers.openai.com/api/docs/guides/model-optimization.