What Is RAG (Retrieval-Augmented Generation) and Why Use It

Retrieval-augmented generation (RAG) is a pattern where your application looks up relevant information from your own data and puts it into the prompt before the language model answers. The model no longer has to rely only on what it learned during training. It reads the facts you hand it and writes an answer from them.

The problem RAG solves

A large language model knows a lot, but three things limit it when you build real products.

• It does not know your data. Your internal wiki, your product docs, your support tickets and your contracts were never in its training set.
• Its knowledge is frozen. Anything that changed after training is missing, and your own documents change every day.
• It can make things up. When the model lacks a fact, it may still produce a confident, fluent, wrong answer.

You could paste every document into every prompt, but that is slow, expensive and often impossible because the data is larger than the context window. RAG fixes this by selecting only the few passages that matter for the current question.

The idea in three steps

1. Index. Split your documents into small pieces called chunks, convert each chunk into a vector (an embedding), and store the vectors in a searchable index.
2. Retrieve. When a user asks a question, convert the question into a vector too, and find the chunks whose vectors are closest to it.
3. Generate. Put the question and the retrieved chunks into a prompt and ask the model to answer using only that material.

The second step is the heart of it. Vector search finds text that is close in meaning, not just text that shares words. A question like "how do I get my money back" can match a paragraph titled "Refund policy" even though they share no keywords.

A tiny example

The code below shows the shape of the idea without any library. It "retrieves" by keyword overlap only, which is a weak stand-in for vector search, but it makes the flow visible: find context, build a prompt, then call the model.

docs = [
    "Refunds are available within 30 days of purchase for annual plans.",
    "Our office is closed on public holidays.",
    "To cancel a subscription, open Settings and choose Billing.",
]

def retrieve(question, k=1):
    words = set(question.lower().split())
    scored = [(len(words & set(d.lower().split())), d) for d in docs]
    scored.sort(reverse=True)
    return [d for _, d in scored[:k]]

question = "Can I get a refund on my annual plan?"
context = "\n".join(retrieve(question))

prompt = (
    "Answer using only the context below.\n\n"
    f"Context:\n{context}\n\nQuestion: {question}"
)
print(prompt)

Running it prints a prompt that contains the refund sentence and the question. Sending that prompt to a model gives an answer grounded in your policy rather than in the model's general knowledge. The rest of this course replaces the toy retrieve function with real embeddings, indexes and ranking.

What RAG gives you

• Fresh answers. Update the index and the answers change. No retraining needed.
• Fewer made-up facts. The model is told to answer from the supplied text, and you can check its claims against that text.
• Citations. Because you know which chunks you supplied, you can show the user their sources.
• Access control. You can filter what is retrieved per user, so people only get answers from documents they may see.
• Lower cost. You send a few hundred relevant tokens instead of a whole knowledge base.

What RAG does not do

RAG is not magic. If retrieval returns the wrong chunks, the model will answer from the wrong material, and a good prompt cannot fully rescue it. Most RAG quality problems are retrieval problems: poor chunking, a weak embedding model, missing keyword matching, or no ranking step. That is why most posts in this course are about the retrieval half.

RAG also does not teach the model a new skill or style. If you need the model to change how it writes or reasons, that is a job for prompting or fine-tuning. We compare the options in a later post.

When to use RAG

• The answers live in documents that you own and that change over time.
• Users need sources they can verify.
• Different users may see different data.
• The knowledge base is too big to fit in one prompt.

If your whole knowledge base is a few pages, you may not need retrieval at all. Put it all in the prompt and keep things simple. Reach for RAG when the data is large, changing or access controlled.

Next steps

Next, read how the full pipeline fits together, then learn what embeddings are. Once you understand those two ideas, every other topic in this course is a way to make one stage of the pipeline faster, cheaper or more accurate.