What Is RAG? Retrieval-Augmented Generation Explained with Code

An LLM only knows what was in its training data. Retrieval-augmented generation (RAG) lets it answer questions about your data, such as docs, tickets or policies, by finding the relevant passages first and putting them in the prompt.

The pipeline

1. Ingest: split documents into chunks and store an embedding (a vector) for each chunk.
2. Retrieve: embed the user's question and find the most similar chunks.
3. Generate: send the question plus the retrieved chunks to the LLM and ask it to answer from them.

A minimal version in Python

import numpy as np
import anthropic
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")  # small local embedding model
client = anthropic.Anthropic()

chunks = [
    "Refunds are available within 30 days of purchase.",
    "Annual plans can be cancelled at any time; unused months are refunded.",
    "Support is available 9am-6pm IST, Monday to Friday.",
]
chunk_vecs = embedder.encode(chunks, normalize_embeddings=True)

def retrieve(question, k=2):
    q = embedder.encode([question], normalize_embeddings=True)[0]
    scores = chunk_vecs @ q                      # cosine similarity (vectors are normalised)
    return [chunks[i] for i in np.argsort(-scores)[:k]]

def answer(question):
    context = "\n".join(f"<doc>{c}</doc>" for c in retrieve(question))
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=16000,
        system="Answer only from the documents provided. If they don't contain the answer, say so.",
        messages=[{"role": "user", "content": f"<docs>\n{context}\n</docs>\n\nQuestion: {question}"}],
    )
    return next(b.text for b in response.content if b.type == "text")

print(answer("Can I get my money back on a yearly plan?"))

Why not just paste everything into the prompt?

• Long context is supported by modern models, but you pay for every token on every request.
• Focused context usually gives more accurate answers than a huge dump of loosely related text.
• RAG scales to millions of documents; a context window doesn't.

If your whole knowledge base fits in a few hundred thousand tokens and rarely changes, try putting it in a cached system prompt first. Reach for RAG when it doesn't fit or changes often.

What makes RAG good or bad

Most RAG failures are retrieval failures: the right chunk was never found. Chunking, hybrid search and reranking (covered in the next articles) matter more than which LLM you use.