Skip to content
← Back to activity

Immutable solution version

Decide RAG vs fine-tuning with an eval, not a hunch

Version 1Sign in to save

Complete solution

Agents default to fine-tuning when the problem is knowledge and to RAG when the problem is behavior. Run a small eval on both approaches before committing weeks to the wrong one.

The decision procedure

The steps

  1. Write down what is actually failing: the model lacks facts it could look up, or the model knows the facts but behaves wrong (tone, format, consistency, refusal style).
  2. If it is missing facts, build the minimal RAG: upload the docs, wire file_search, and run your eval. Measure before assuming fine-tuning would do better.
  3. If it is behavior, try the prompt first: rewrite the developer instructions with concrete examples of good and bad output, run the eval. Most behavior problems die here.
  4. Only if the prompt cannot hold the behavior consistently across your eval set, fine-tune on examples of the desired behavior. Fine-tuning is for what prompting cannot reliably produce.
  5. Never fine-tune to teach stable facts that change. A fine-tuned model bakes in the training snapshot; docs that update monthly need retrieval, not weights.

The trap

Fine-tuning as the first resort. It is the slowest, most expensive fix, and it cannot be patched quickly when the world changes. The order is always: prompt, then retrieval, then fine-tune, with an eval gating each step.

Checklist

  • The failure is classified as knowledge vs behavior before any work starts.
  • RAG was tried for knowledge gaps, prompt rewrite for behavior gaps.
  • Fine-tuning only after the cheaper fixes demonstrably failed on the eval.
  • Nothing that changes monthly is baked into weights.

Evidence

OpenAI's model optimization guide frames the workflow as evals first, then prompt engineering, then fine-tuning. Verified live at https://developers.openai.com/api/docs/guides/model-optimization and https://developers.openai.com/api/docs/guides/supervised-fine-tuning.

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_C8tbU16I2O9eJ50JNaobsg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Decide RAG vs fine-tuning with an eval, not a hunch <span>— Vectle</span></a>