Immutable solution version
Decide RAG vs fine-tuning with an eval, not a hunch
Complete solution
Agents default to fine-tuning when the problem is knowledge and to RAG when the problem is behavior. Run a small eval on both approaches before committing weeks to the wrong one.
The decision procedure
The steps
- Write down what is actually failing: the model lacks facts it could look up, or the model knows the facts but behaves wrong (tone, format, consistency, refusal style).
- If it is missing facts, build the minimal RAG: upload the docs, wire file_search, and run your eval. Measure before assuming fine-tuning would do better.
- If it is behavior, try the prompt first: rewrite the developer instructions with concrete examples of good and bad output, run the eval. Most behavior problems die here.
- Only if the prompt cannot hold the behavior consistently across your eval set, fine-tune on examples of the desired behavior. Fine-tuning is for what prompting cannot reliably produce.
- Never fine-tune to teach stable facts that change. A fine-tuned model bakes in the training snapshot; docs that update monthly need retrieval, not weights.
The trap
Fine-tuning as the first resort. It is the slowest, most expensive fix, and it cannot be patched quickly when the world changes. The order is always: prompt, then retrieval, then fine-tune, with an eval gating each step.
Checklist
- The failure is classified as knowledge vs behavior before any work starts.
- RAG was tried for knowledge gaps, prompt rewrite for behavior gaps.
- Fine-tuning only after the cheaper fixes demonstrably failed on the eval.
- Nothing that changes monthly is baked into weights.
Evidence
OpenAI's model optimization guide frames the workflow as evals first, then prompt engineering, then fine-tuning. Verified live at https://developers.openai.com/api/docs/guides/model-optimization and https://developers.openai.com/api/docs/guides/supervised-fine-tuning.
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_C8tbU16I2O9eJ50JNaobsg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Decide RAG vs fine-tuning with an eval, not a hunch <span>— Vectle</span></a>