RAG vs Fine-Tuning vs Long Context: Which Should You Use?

Use RAG when the model needs access to facts that live in your documents, use fine-tuning when you need to change how the model behaves, and use a long context window when the whole knowledge base fits in one prompt. These options solve different problems, and many strong systems combine two of them.

Three ways to give a model your knowledge

Long context

Modern models accept very large prompts. The simplest design is to paste all your documents into the prompt and ask the question. There is no index, no chunking and nothing to keep in sync. The model can reason across the whole text, which helps with questions like "summarize the differences between these two contracts."

The downsides are cost and speed. You pay for every input token on every request, and large prompts take longer to process. Accuracy can also drop when a prompt holds a lot of irrelevant text. Prompt caching reduces the cost of repeated prefixes, but the knowledge base still has to fit.

RAG

RAG keeps knowledge outside the model and brings in only the relevant passages for each question. It scales to millions of documents, updates the moment you re-index, supports citations and lets you filter by user permissions. The cost is engineering: you must build and tune ingestion and retrieval.

Fine-tuning

Fine-tuning changes the model's weights using examples. It is good at teaching a consistent style, a fixed output format or a narrow task. It is a poor way to teach facts: the knowledge is hard to update, hard to verify, impossible to cite and can still be recalled wrongly. It also cannot enforce who may see which fact.

Comparison

Question	Long context	RAG	Fine-tuning
Data changes daily	Easy	Easy	Hard
Data larger than the context window	Not possible	Designed for it	Possible but lossy
Need citations	Possible	Natural fit	Not possible
Per-user access rules	You must filter before prompting	Metadata filters	Not possible
Cost per query	High for large data	Low to moderate	Low
Change tone or format	Prompting helps	Prompting helps	Best fit
Setup effort	Lowest	Moderate	Highest

A decision rule

1. Does all the knowledge fit in the prompt, and is the request volume low or cacheable? Start with long context.
2. Is the knowledge large, changing, citable or permissioned? Use RAG.
3. Is the problem the model's behavior (format, tone, a narrow classification skill) rather than missing facts? Consider fine-tuning, after you have tried better prompts.
4. Do you have both a knowledge need and a behavior need? Combine them: RAG for facts, fine-tuning or prompting for style.

A quick size check in code

To decide whether long context is viable, count the tokens in your corpus. The Claude SDK includes a token-counting call. This script totals a folder of text files and prints the result so you can compare it with the context window of the model you plan to use.

import pathlib
import anthropic

client = anthropic.Anthropic()

def count_tokens(text: str) -> int:
    result = client.messages.count_tokens(
        model="claude-sonnet-5-5",
        messages=[{"role": "user", "content": text}],
    )
    return result.input_tokens

total = 0
for path in pathlib.Path("docs").glob("**/*.txt"):
    total += count_tokens(path.read_text(encoding="utf-8"))

print(f"Corpus size: {total:,} tokens")

If the total is a small fraction of the window, long context is a real option. If it is many times larger, or it grows every week, plan for retrieval. Also multiply the size by your expected request volume to estimate the bill before you commit.

Common mistakes

• Fine-tuning to inject facts. Teams fine-tune on their wiki and then find the model still invents details and cannot show a source. Use RAG for facts.
• Building RAG for a tiny corpus. A 20-page handbook does not need a vector database. Put it in the prompt.
• Assuming a bigger window removes the need for retrieval. Large windows help, but cost, latency, permissions and freshness still push big systems toward retrieval.
• Treating the choice as permanent. You can start with long context for a prototype and move to RAG when the data grows.

Pick the simplest option that meets your needs today, measure it, and add complexity only when a measurement says you must. The remaining posts assume you chose RAG and want to do it well.