fine tune vs prompt cost comparison error wrong recommendation fails

Not specified by the original author.

TL;DR

If it is missing facts, build the minimal RAG: upload the docs, wire file_search, and run your eval. If it is behavior, try the prompt first: rewrite the developer instructions with concrete examples of good and bad output, run the eval.

Steps

1. Write down what is actually failing: the model lacks facts it could look up, or the model knows the facts but behaves wrong (tone, format, consistency, refusal style).
2. If it is missing facts, build the minimal RAG: upload the docs, wire file_search, and run your eval. Measure before assuming fine-tuning would do better.
3. If it is behavior, try the prompt first: rewrite the developer instructions with concrete examples of good and bad output, run the eval. Most behavior problems die here.
4. Only if the prompt cannot hold the behavior consistently across your eval set, fine-tune on examples of the desired behavior. Fine-tuning is for what prompting cannot reliably produce.
5. Never fine-tune to teach stable facts that change. A fine-tuned model bakes in the training snapshot; docs that update monthly need retrieval, not weights.
6. Fine-tuning as the first resort. It is the slowest, most expensive fix, and it cannot be patched quickly when the world changes. The order is always: prompt, then retrieval, then fine-tune, with an eval gating each step.
7. - The failure is classified as knowledge vs behavior before any work starts.

• RAG was tried for knowledge gaps, prompt rewrite for behavior gaps.
• Fine-tuning only after the cheaper fixes demonstrably failed on the eval.
• Nothing that changes monthly is baked into weights.

When to use

You are seeing this: Write down what is actually failing: the model lacks facts it could look up, or the model knows the facts but behaves wrong (tone, format, consistency, refusal style). Use this skill when you run into "Decide RAG vs fine-tuning with an eval, not a hunch".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

OpenAI's model optimization guide frames the workflow as evals first, then prompt engineering, then fine-tuning. Verified live at https://developers.openai.com/api/docs/guides/model-optimization and https://developers.openai.com/api/docs/guides/supervised-fine-tuning.

Not specified by the original author.

TL;DR: A failed fine-tune job almost always means the training file was malformed. Validate the JSONL locally, read the job events for the exact error, and start small before relaunching the full run. Each blind retry burns money, so never relaunch without knowing the cause. Confirm the cause Validate the training file locally before it touches the API: 1.

The error

. A typo in a role name fails the whole file.
3. **Upload purpose.** The file must be uploaded with purpose

The symptom

The fine-tuning job errors out or sits in failed state. Each blind retry burns money, so never relaunch without knowing the cause.

Confirm the cause

Validate the training file locally before it touches the API:

1. JSONL shape. One complete JSON object per line. No pretty-printed multi-line objects, no trailing commas.
2. Message format. Each line needs a messages array, every message with role and content. A typo in a role name fails the whole file.
3. Upload purpose. The file must be uploaded with purpose fine-tune. A file uploaded for another purpose is not accepted as training data.
4. Size sanity. Check token counts per example. Pathological examples turn a validation problem into a billing problem.

Then read the job's events, not just its status. The events carry the specific validation error: which line, which field. Fix that exact thing.

The fix

1. Write a local validator: every line parses as JSON, has a messages array, every message has role and content, no empty strings where forbidden.
2. Fix all errors, re-upload with purpose fine-tune, launch a new job.
3. Start with a few hundred examples to prove the pipeline before training on the full dataset.
4. Estimate cost from token counts before the full run.

Verify the fix

The validator passes clean on the exact file uploaded. The new job reaches running state with steady progress in its events, not a validation error. Keep the validator in the pipeline so the next dataset is checked automatically.

When to use this

• You are seeing this exact error message; match the block above, not just part of it.
• The failing call matches the scenario in the title: OpenAI fine-tune job failed.
• You want the fastest verified fix before digging through logs.

When not to use this

• Your error text differs from the block above; close cousins often have different causes.
• The stack trace points at a different component than the one in the title.
• You already applied this fix and the error persists; look for a second cause instead of reapplying.

Compatibility

• Not pinned to a specific version; follows current OpenAI behavior.

OpenAI's fine-tuning guide requires the dataset uploaded formatted in JSONL with purpose fine-tune, and job events surface the specific validation errors. Local validation of line shape and message format before launching avoids paying for repeated failed jobs (https://platform.openai.com/docs/guides/fine-tuning).

Not specified by the original author.

TL;DR

Use code where you can: exact match, regex, required tool called, required file exists. Most use cases need evaluation along several success criteria: correctness, citation accuracy, tone, latency, cost per task. Use an LLM judge only where code cannot reach (tone, nuance, open-ended quality). When you use a judge, recalibrate it every time you change the model under test, judges drift with the model.

When to use

You are seeing this: Do this before writing or changing any prompt. Use this skill when you run into "Define success criteria before you tune a single Claude prompt".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Do this before writing or changing any prompt.

1. Write success criteria as sentences you can measure. Good criteria are specific, measurable, achievable, and relevant. "Good performance" is not a criterion. "The refund answer cites the correct policy section in 95 percent of cases" is.
2. Go multidimensional. Most use cases need evaluation along several success criteria: correctness, citation accuracy, tone, latency, cost per task. One number hides regressions in the others.
3. Build cases from real failures, bottom-up. Collect 20 to 50 real inputs where the current behavior was wrong or borderline, including edge cases and prompt-injection attempts if the prompt handles external content. Design evals that mirror your real-world task distribution, not clean textbook examples.
4. Choose graders per criterion. Use code where you can: exact match, regex, required tool called, required file exists. Use an LLM judge only where code cannot reach (tone, nuance, open-ended quality). When you use a judge, recalibrate it every time you change the model under test, judges drift with the model.
5. Prefer volume over perfect labels. Automate when possible and accept noisy labels at volume over a handful of hand-perfect cases. A 200-case automated suite with 90 percent label accuracy beats a 10-case perfect one for catching regressions.
6. Freeze the set before tuning. Lock the cases and graders, record a baseline score, then start changing prompts. Any change to the eval set after tuning starts must be logged as a new version, or you will tune to a moving target.

Claims verified against Anthropic's official docs (platform.claude.com) on 2026-09-26. Evidence URLs in public_references.