How to Choose an Embedding Model for RAG

Choose an embedding model by testing candidates on your own questions and documents, then weigh quality against cost, speed, vector size and hosting. Public leaderboards are a good way to shortlist models, but they are an average over other people's data. Your data decides the winner.

The criteria that matter

Retrieval quality on your content

This is the main criterion. Technical docs, legal text, support chats and code behave differently. A model that tops a general benchmark can lose on your domain vocabulary. Measure it with a small evaluation set, which a later post in this course shows how to build.

Vector dimensions

Dimensions are the length of each vector. More dimensions can hold more detail but cost more memory, disk and search time. A 1,536-dimension float32 vector takes about 6 KB, so ten million chunks need roughly 60 GB for the vectors alone, before index overhead. A 384-dimension model needs a quarter of that. Some models let you shorten their vectors with little quality loss, which we cover in the Matryoshka post.

Maximum input length

Each model reads a limited number of tokens. If your chunks are 500 tokens and the model reads only 256, half of every chunk is ignored. Check the limit and match your chunk size to it.

Languages

If your users or documents use more than one language, use a multilingual model. English-only models degrade quickly on other languages, and cross-language search (an English question matching a German document) needs a model trained for it.

Cost and speed

Hosted models charge per token embedded. Local models cost compute instead. Estimate the one-time cost of embedding your corpus, the recurring cost of embedding new content, and the per-query cost. Query embedding is usually tiny compared with the language model call, but it adds latency if the API is far away.

Hosting and data policy

Sending documents to a third-party API may be forbidden for sensitive data. Open models you run yourself keep data inside your network. Check retention and training policies for any hosted service.

Lock-in

Changing models means re-embedding everything, because vectors from different models cannot be mixed. Design so that it is possible: record the model name and version in your index metadata and keep the original text so you can rebuild.

A simple shortlist process

1. Pick three to five candidates: one small local model, one or two strong hosted models, and one multilingual model if you need it.
2. Write 50 to 100 realistic questions and mark which chunk or document answers each.
3. Embed your corpus with each candidate.
4. Measure how often the right chunk appears in the top 5 (recall at 5).
5. Compare the quality gain with the cost, latency and storage differences.

A small comparison harness

The function below scores any model on a list of question and answer-chunk pairs. It uses sentence-transformers so you can try several local models by name. The same structure works for hosted models by swapping the encode call.

import numpy as np
from sentence_transformers import SentenceTransformer

def recall_at_k(model_name, chunks, qa_pairs, k=5):
    """qa_pairs: list of (question, index_of_correct_chunk)."""
    model = SentenceTransformer(model_name)
    chunk_vecs = model.encode(chunks, normalize_embeddings=True)
    hits = 0
    for question, correct in qa_pairs:
        q = model.encode([question], normalize_embeddings=True)[0]
        top = np.argsort(-(chunk_vecs @ q))[:k]
        hits += int(correct in top)
    return hits / len(qa_pairs)

# Example call (use your own data):
# for name in ["all-MiniLM-L6-v2", "paraphrase-multilingual-MiniLM-L12-v2"]:
#     print(name, recall_at_k(name, chunks, qa_pairs))

Run it once per candidate and compare the numbers. Keep the evaluation set so you can rerun it when a new model appears.

Practical details people forget

Run the shortlist on the hardware and network path you will use in production. A model that looks fast on a developer laptop may be slow on a small container, and a hosted model may add a network round trip to every query. Measure the time to embed one query, because that delay sits directly in front of the user's wait. Also measure batch throughput for ingestion, since embedding a large corpus can take hours with a slow model.

Consider the long-term picture too. Will the provider keep the model available? Will you be able to pin a version so results do not change under you? Does the license allow your use? These questions are boring but they decide whether a choice that works today is still working in two years.

Finally, remember that the embedding model is only one piece. A modest model combined with keyword search and a reranker often beats a premium model used alone. Do not spend a large budget on the model before you have tried the cheaper retrieval improvements described later in this course.

Common mistakes

• Choosing by leaderboard rank alone without testing on your own data.
• Ignoring the input limit, so chunks are silently truncated.
• Forgetting prefixes or special instructions that a model requires for queries or documents.
• Picking the largest vectors "to be safe" and then paying for memory you do not need.
• Not recording which model produced the stored vectors.

A model that is good enough and cheap to run beats a perfect model you cannot afford to re-embed. Start small, measure, and upgrade only if the numbers justify it.