Fine-Tuning vs Prompting vs RAG: How to Choose

Use prompting first, add retrieval when the model needs facts it does not have, and fine-tune when you need a consistent behaviour, format or cost profile that prompting cannot deliver. The three techniques solve different problems, and many production systems combine them.

What each technique changes

• Prompting changes the input. You give instructions, examples and context. Nothing about the model changes, so you can iterate in minutes.
• RAG (retrieval-augmented generation) changes the input too, but it fetches relevant documents at request time and places them in the prompt. It solves missing or changing knowledge.
• Fine-tuning changes the model. It solves missing behaviour: format, style, task skill, or the need to do the same job with a smaller model.

A decision table

Your problem	Try first	Why
Answers must use private or fresh documents	RAG	Facts live in an index you can update without training
Output format is sometimes wrong	Prompting with examples, then structured output features	Cheap to test; fine-tune if errors remain
Need a specific brand voice at scale	Fine-tuning	Style is hard to hold with prompts alone
Prompt is huge and costly per request	Fine-tuning	Bake instructions into weights, send shorter prompts
Latency or cost needs a small model	Fine-tuning or distillation	Small models need task-specific training to compete
Facts change weekly	RAG	Retraining each week is slow and error-prone

The escalation ladder

Climb in order, and stop as soon as the quality target is met:

1. Write a clear prompt with a few examples. Measure it on a fixed test set.
2. Improve the prompt: split the task, add constraints, use structured output.
3. Add retrieval if failures are caused by missing information.
4. Fine-tune if failures are caused by behaviour, and you have good examples.
5. Combine: a fine-tuned model that also receives retrieved context.

Each step costs more time and money than the one before. Skipping steps is the most common waste in this field.

Diagnose the failure first

Look at ten to thirty bad outputs and sort them. Ask: did the model lack information, or did it have the information and behave badly? If the correct answer was not in the prompt and the model could not know it, that is a knowledge failure, so use RAG. If the correct information was in the prompt and the model still ignored the format, rambled or used the wrong tone, that is a behaviour failure, so consider fine-tuning.

Here is a tiny script that makes the sorting concrete by tagging each failure in a spreadsheet-like structure and counting categories:

from collections import Counter

failures = [
    {"id": 1, "cause": "missing_knowledge"},
    {"id": 2, "cause": "wrong_format"},
    {"id": 3, "cause": "wrong_format"},
    {"id": 4, "cause": "wrong_tone"},
    {"id": 5, "cause": "missing_knowledge"},
    {"id": 6, "cause": "wrong_format"},
]

counts = Counter(f["cause"] for f in failures)
for cause, n in counts.most_common():
    print(f"{cause}: {n}")
# wrong_format: 3 -> candidate for fine-tuning
# missing_knowledge: 2 -> candidate for RAG

The output lists causes by frequency. Whichever dominates tells you where to invest.

Combining the techniques

These are not rivals. A common pattern is to fine-tune a small model to follow a strict answer format and to cite its sources, then feed it retrieved passages at run time. The weights supply behaviour, and the retrieval supplies facts. Another pattern is to fine-tune the retriever's embedding model, which is a separate topic from tuning a generator.

Cost and speed comparison

• Prompting: no training cost, but longer prompts cost more per request.
• RAG: needs an index and a retrieval step, adds latency, but updates are instant.
• Fine-tuning: a one-time training cost plus evaluation effort, then shorter prompts and sometimes a cheaper model.

Checklist

• Do you have a fixed test set and a baseline score from prompting alone?
• Are the failures behavioural rather than missing facts?
• Do you have at least a few hundred good examples, or a way to make them?
• Can you live with retraining when requirements change?

If you answered no to the first question, stop and build the baseline. Post 16 shows how.