Reranking & Hybrid Search

Why embeddings alone aren't enough

RAG retrieval starts with embeddings: embed the query, find the k nearest vectors, return those chunks. It works well for conceptual queries — "how do I handle authentication errors?" maps to a passage about OAuth token refresh even if the words don't overlap.

But embeddings fail on exact-match queries:

• Error codes: "ERR_CONNECTION_REFUSED" — the embedding model doesn't treat this as special.
• Product SKUs: "SKU-48291-B" — meaningless to a semantic model.
• Person names: "What did Sarah Chen say about the migration?" — embeddings conflate similar names.
• Legal citations: "Section 4.2.1(a)" — precision matters, semantics don't help.

BM25 (Best Matching 25) is the classic keyword search algorithm. It scores documents by term frequency and inverse document frequency — exact word matches, no neural network. It's fast, interpretable, and catches what embeddings miss.

Hybrid search runs both and merges the results. Then a reranker re-scores the merged candidates with a more expensive but more accurate model.

The three-layer retrieval stack

Layer 1: Candidate generation (fast, recall-oriented)
  ├── Dense retrieval (embeddings) → top-50
  └── Sparse retrieval (BM25)      → top-50

Layer 2: Fusion
  └── Reciprocal Rank Fusion (RRF) → merged top-50

Layer 3: Reranking (slow, precision-oriented)
  └── Cross-encoder reranker → top-5 for LLM context

Each layer has a different job. Candidate generation casts a wide net. Fusion deduplicates and combines. Reranking picks the best.

Layer 1: Dual retrieval

Dense retrieval (embeddings)

Embed the query, cosine-similarity search against your vector database, return top-50.

Strengths: semantic similarity, paraphrase matching, cross-language (with multilingual models). Weaknesses: exact terms, rare tokens, numbers, proper nouns.

Sparse retrieval (BM25)

Tokenize the query, score documents by term overlap with TF-IDF weighting, return top-50.

Strengths: exact matches, rare terms, numbers, codes. Weaknesses: no semantic understanding ("car" won't match "automobile").

Running both

You don't need Elasticsearch. Common setups:

The two indexes can live in the same database or separate services. What matters is that both run on every query.

Layer 2: Reciprocal Rank Fusion

You have two ranked lists. How do you merge them?

Don't normalize and weight the scores — embedding similarity scores and BM25 scores live on different scales and weighting them is fragile.

Do use Reciprocal Rank Fusion:

RRF_score(doc) = Σ  1 / (k + rank_i)

For each document, sum the reciprocal rank across all retrieval lists. A document ranked #1 in BM25 and #3 in embeddings scores higher than one ranked #10 in both.

k = 60 is the standard constant (from the original paper). It dampens the impact of very high ranks so a #1 in one list doesn't dominate.

RRF is parameter-free, robust, and consistently beats hand-tuned score blending. Use it unless you have a strong reason not to.

Layer 3: Cross-encoder reranking

Candidate generation is fast but imprecise — it optimizes for recall (don't miss the right doc). Reranking is slower but precise — it optimizes for precision (rank the best doc first).

A cross-encoder takes the query and each candidate chunk as a pair and outputs a relevance score. Unlike bi-encoders (embeddings), the cross-encoder sees both texts together, so it catches subtle relevance signals.

Query: "How do I rotate API keys?"
Candidate 1: "...navigate to Settings > API Keys..."     → score: 0.94
Candidate 2: "...API keys are used for authentication..." → score: 0.71
Candidate 3: "...key rotation policy requires approval..." → score: 0.45

Reranker options

For most teams: start with Cohere rerank-3 (simple API, strong quality). Move to self-hosted when cost or latency at scale demands it.

How many to rerank?

• Retrieve 50, rerank to 5 — the standard production config.
• Fewer than 30 candidates: you miss good docs that ranked #35 in one list.
• More than 100: latency grows linearly, quality gains flatten.

Putting it together — a complete query

# Pseudocode for a hybrid + rerank retrieval pipeline

def retrieve(query: str, top_k: int = 5) -> list[Chunk]:
    # Layer 1: dual retrieval
    dense_results = vector_db.search(embed(query), top_k=50)
    sparse_results = bm25_index.search(query, top_k=50)

    # Layer 2: fusion
    merged = reciprocal_rank_fusion(dense_results, sparse_results, k=60)

    # Layer 3: reranking
    reranked = reranker.rerank(query, merged[:50], top_k=top_k)

    return reranked

Total latency: ~50ms retrieval + ~30ms reranking = ~80ms. Compare to 1–5 seconds for LLM generation. The retrieval stack is never your bottleneck.

Measuring retrieval quality

Don't ship hybrid search without measuring it. Use the same eval framework you'd use for chunking:

1. Build a retrieval eval set — questions with known gold-source passages.
2. Measure recall@k — is the gold passage in the top-k results?
3. Compare configurations:

4. Inspect failures — when recall drops, check if it's a chunking issue (chunking strategies) or a retrieval issue.

Run this eval when you change embedding models, add hybrid search, or switch rerankers. Retrieval regressions are the most common cause of RAG quality drops.

When to add each layer

Don't over-engineer v0. Add layers when evals show you need them.

Common failure modes

• Reranking the entire index — cross-encoders don't scale to millions of docs. Retrieve first, rerank second.
• Tuning BM25/dense weights manually — use RRF instead. It's more robust and requires zero tuning.
• Different tokenization at index vs query — BM25 is sensitive to stemming and stop-word handling. Use the same tokenizer for both.
• Ignoring chunking — hybrid search retrieves better chunks, but if chunking is broken, better retrieval just finds broken chunks faster.
• No eval baseline — you add reranking, answers feel better, but you can't prove it or catch regressions.

The bottom line

Embeddings capture meaning. BM25 captures exact terms. Reranking captures relevance. Together they form the retrieval stack that production RAG systems actually use.

Start with embeddings. Add BM25 when evals show exact-match failures. Add reranking when you need the last mile of precision. Measure everything. The embedding model comparison matters less than getting this stack right.