Reranking Search Results to Improve RAG Accuracy

Embedding search compares a query vector with chunk vectors that were computed separately. It is fast, but it misses nuance. A reranker reads the query and each candidate chunk together and scores how well they match. It is slower, so you use it on a short list.

Two-stage retrieval

1. Retrieve a generous candidate set (for example top 50) with vector or hybrid search.
2. Rerank those 50 with a cross-encoder.
3. Send only the top 5-10 to the LLM.

Reranking with a cross-encoder

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query, candidates, top_n=5):
    pairs = [(query, c["content"]) for c in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [c for c, _ in ranked[:top_n]]

candidates = hybrid_search(query, limit=50)
context_chunks = rerank(query, candidates)

Hosted rerankers

Several providers (for example Cohere and Voyage AI) offer rerank APIs. They take a query and a list of documents and return relevance scores, so you get a strong model without running one yourself.

Using an LLM as a reranker

For small candidate sets you can ask an LLM to pick the relevant chunks, ideally with structured output listing chunk IDs. It's more expensive per query but needs no extra model.

Is it worth it?

• Reranking typically raises the share of questions where the right chunk is in the final context, which directly improves answers.
• It adds latency (tens to hundreds of milliseconds). Rerank fewer candidates if that matters.
• Measure with an evaluation set before and after. Keep it only if it helps on your data.