Pick the OpenAI model with evals: smallest that passes, reasoning where it pays

Model selection as a procedure

The steps

1. Build the eval first, before choosing a model. Without a pass/fail bar, "the big model feels better" is not a decision.
2. Start with the smallest, cheapest model that could plausibly do the job. Run the full eval. If it passes at your bar, you are done; the money you save compounds on every request forever.
3. Step up one tier only on measured failure. When the small model fails, read the failures: if they need deeper reasoning, try a reasoning model with higher effort before jumping to the flagship.
4. Match reasoning effort to the task. Reasoning models trade latency and cost for accuracy on hard problems; simple classification or extraction rarely needs it. Do not pay reasoning prices for non-reasoning work.
5. Re-run the eval on every model change, including minor version bumps. Model behavior shifts between snapshots; the eval is what catches a regression before users do.
6. Keep a fallback model configured. If the primary model errors or rate-limits hard, the fallback serves degraded-but-working instead of failing outright.

The trap

Standardizing on the flagship "to be safe." It is safe for quality and unsafe for cost and latency. Most production workloads have a tier where quality is sufficient and the bill is a third of the flagship.

Checklist

• Eval exists before the model is chosen.
• Smallest passing model is the default.
• Step-ups are justified by eval failures, not vibes.
• Reasoning effort matched to task difficulty.
• Eval re-runs on every model version change; fallback configured.

OpenAI's model optimization guide frames evals as the measurement layer for model and prompt choices, and the web search guide documents reasoning levels (high, xhigh) for hard tasks. Verified live at https://developers.openai.com/api/docs/guides/model-optimization.