Decide RAG vs fine-tuning with an eval, not a hunch

TL;DR

If it is missing facts, build the minimal RAG: upload the docs, wire file_search, and run your eval. If it is behavior, try the prompt first: rewrite the developer instructions with concrete examples of good and bad output, run the eval.

Steps

1. Write down what is actually failing: the model lacks facts it could look up, or the model knows the facts but behaves wrong (tone, format, consistency, refusal style).
2. If it is missing facts, build the minimal RAG: upload the docs, wire file_search, and run your eval. Measure before assuming fine-tuning would do better.
3. If it is behavior, try the prompt first: rewrite the developer instructions with concrete examples of good and bad output, run the eval. Most behavior problems die here.
4. Only if the prompt cannot hold the behavior consistently across your eval set, fine-tune on examples of the desired behavior. Fine-tuning is for what prompting cannot reliably produce.
5. Never fine-tune to teach stable facts that change. A fine-tuned model bakes in the training snapshot; docs that update monthly need retrieval, not weights.
6. Fine-tuning as the first resort. It is the slowest, most expensive fix, and it cannot be patched quickly when the world changes. The order is always: prompt, then retrieval, then fine-tune, with an eval gating each step.
7. - The failure is classified as knowledge vs behavior before any work starts.

• RAG was tried for knowledge gaps, prompt rewrite for behavior gaps.
• Fine-tuning only after the cheaper fixes demonstrably failed on the eval.
• Nothing that changes monthly is baked into weights.

When to use

You are seeing this: Write down what is actually failing: the model lacks facts it could look up, or the model knows the facts but behaves wrong (tone, format, consistency, refusal style). Use this skill when you run into "Decide RAG vs fine-tuning with an eval, not a hunch".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

OpenAI's model optimization guide frames the workflow as evals first, then prompt engineering, then fine-tuning. Verified live at https://developers.openai.com/api/docs/guides/model-optimization and https://developers.openai.com/api/docs/guides/supervised-fine-tuning.