Rerankers and Hybrid Search: Getting Retrieval Right in RAG

A team ships a support assistant on top of a vector database. Demos go well. Then a user pastes an error code, “ERR_4012”, and the assistant answers confidently about something unrelated. The right document exists in the index. The embedding model simply did not treat a string of letters and digits as meaningful. This is the most common way retrieval-augmented generation fails: the language model is fine, but the retrieval step handed it the wrong passages. This article covers the two upgrades that fix most of these failures, hybrid search and reranking, and shows how to build, tune and evaluate them.

Why Retrieval Is the Bottleneck

In a RAG system, the generator can only be as good as the context it receives. If the relevant passage is not in the top results, no prompt will rescue the answer. Retrieval has two competing goals:

• Recall: make sure the right passage is somewhere in the candidate set.
• Precision: make sure the best passages are at the top, because context windows are limited and irrelevant text distracts the model.

A single retrieval method rarely does both well. The standard solution is a pipeline: cast a wide net cheaply, then rank the catch carefully.

Dense Retrieval and Its Blind Spots

Dense retrieval encodes the query and every chunk into vectors with a bi-encoder, then finds the nearest neighbors. See embeddings for the background. It is excellent at paraphrase and meaning: “how do I cancel my plan” matches “ending your subscription.”

It is weaker in predictable places:

• Exact identifiers. Error codes, SKUs, function names, ticket numbers and legal clause numbers carry little semantic content, and embeddings blur them.
• Rare or new terminology. Terms absent from the embedding model’s training data get poor representations.
• Negation and precise constraints. “Without a credit card” and “with a credit card” can land close together.
• Compression. A whole chunk is squeezed into one vector. Fine detail is lost.

Lexical Search: BM25

BM25 is the classic keyword ranking function. It scores a document by how many query terms it contains, weighted by how rare each term is across the corpus and normalized for document length. It knows nothing about meaning, and that is precisely why it complements embeddings: it nails exact tokens like “ERR_4012” and rare words, and it fails on synonyms.

Aspect	BM25 (lexical)	Dense (semantic)
Strength	Exact terms, identifiers, rare words	Paraphrase, intent, cross-language
Weakness	Vocabulary mismatch	Identifiers, precise constraints
Index cost	Cheap, no model needed	Needs an embedding model
Explainability	Easy to inspect	Opaque

Running both and merging the results is called hybrid search.

Fusing Results: Reciprocal Rank Fusion

The two retrievers produce scores on incompatible scales: BM25 scores are unbounded, cosine similarities lie between -1 and 1. Normalizing and weighting them is fiddly. A robust alternative that ignores scores entirely is reciprocal rank fusion (RRF). Each document earns 1 / (k + rank) from every list it appears in, and the contributions are summed.

from collections import defaultdict

def rrf(rankings, k=60):
    """rankings: list of lists of doc ids, best first."""
    scores = defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

bm25_ids  = ["d7", "d2", "d9", "d1"]
dense_ids = ["d2", "d5", "d7", "d3"]
print(rrf([bm25_ids, dense_ids]))

A document that appears near the top of both lists beats one that is first in only one. The constant k (60 is the commonly used default) dampens the influence of the very top ranks. RRF needs no tuning to get started, which is why many search engines and vector databases offer it as the default fusion method.

A minimal hybrid retriever using the rank_bm25 and sentence-transformers libraries:

from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
import numpy as np

docs = [
    "Error ERR_4012 means the API key has expired. Rotate it in settings.",
    "To end your subscription, open Billing and choose Cancel plan.",
    "Rotate credentials regularly to limit the impact of a leak.",
]

tokenized = [d.lower().split() for d in docs]
bm25 = BM25Okapi(tokenized)

encoder = SentenceTransformer("all-MiniLM-L6-v2")
doc_vecs = encoder.encode(docs, normalize_embeddings=True)

def hybrid_search(query, top_k=3):
    bm25_scores = bm25.get_scores(query.lower().split())
    bm25_rank = list(np.argsort(-bm25_scores))

    q_vec = encoder.encode([query], normalize_embeddings=True)[0]
    dense_rank = list(np.argsort(-(doc_vecs @ q_vec)))

    return rrf([bm25_rank, dense_rank])[:top_k]

print(hybrid_search("what does ERR_4012 mean"))

Real systems use a proper inverted index (Elasticsearch, OpenSearch, Postgres full-text search, or a vector database that supports sparse vectors) rather than scoring every document in Python, but the logic is the same. See vector databases for the storage side.

Why You Still Need a Reranker

Hybrid search improves recall, but the ordering inside the candidate list is still rough, because both retrievers score the query and each document independently. A bi-encoder never sees the query and the document together.

A cross-encoder does. It feeds the query and one candidate through a transformer jointly, so attention can compare every query word with every document word, and outputs a relevance score. That joint view is far more accurate, and far more expensive: you cannot precompute anything, and you pay one model forward pass per query-document pair. That is why cross-encoders are used only on a short list.

The resulting two-stage pattern is the workhorse of modern search:

1. Retrieve a generous candidate set (for example, the top 50 to 100) with hybrid search.
2. Rerank those candidates with a cross-encoder.
3. Keep the top handful (for example 3 to 8) as context for the generator.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def retrieve_and_rerank(query, candidates, keep=3):
    pairs = [(query, c) for c in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [text for text, _ in ranked[:keep]]

The checkpoint name above is a widely used open model; swap in whichever reranker fits your language and domain. Hosted rerank APIs from several vendors work the same way: you send a query and a list of passages and get back relevance scores.

An alternative is using an LLM as the reranker, asking it to score or order passages. This can be strong but is slower and costlier, and it needs care around position bias and output formatting. It is worth trying when you have a small candidate set and a quality problem that cross-encoders do not solve.

Choosing Candidate and Final Sizes

Two numbers control the pipeline:

• Candidate count. Larger improves recall but costs reranker latency in direct proportion. If the right document is not in the candidates, reranking cannot recover it.
• Final count. Smaller keeps the prompt focused and cheap; larger helps questions that need several passages.

Measure rather than guess. Plot recall against candidate count and pick the knee of the curve.

Chunking Still Matters

Reranking does not repair bad chunks. If a chunk cuts a table in half, or separates a heading from its content, both retrievers and the reranker suffer. Prefer chunks that follow document structure, keep titles attached, and consider storing a short context prefix (document title, section name) with each chunk. Many cross-encoders also have a maximum input length, so chunks longer than that limit are silently truncated.

Evaluating Retrieval Separately

The most valuable habit is to evaluate retrieval on its own, before the generator is involved. Build a small set of real questions, each labeled with the passages that answer it. Then compute:

• Recall@k: the fraction of questions where a relevant passage appears in the top k.
• MRR (mean reciprocal rank): the average of 1/rank of the first relevant result.
• nDCG: a rank-aware score that handles graded relevance.

def recall_at_k(results, relevant, k):
    hits = sum(1 for q, ids in results.items() if set(ids[:k]) & relevant[q])
    return hits / len(results)

def mrr(results, relevant):
    total = 0.0
    for q, ids in results.items():
        for rank, doc_id in enumerate(ids, start=1):
            if doc_id in relevant[q]:
                total += 1.0 / rank
                break
    return total / len(results)

Run the same questions through dense only, BM25 only, hybrid, and hybrid plus reranker. The comparison tells you which stage earns its cost on your data. Even a few dozen labeled questions are far better than intuition; see evaluating LLM applications for how to grow this into a proper suite.

When Not to Add These Layers

Complexity has a price, so skip stages that do not pay for themselves:

• A small corpus (a few hundred short documents) often works with dense search alone, or even by placing the material directly in the prompt.
• If your queries are all natural-language questions with no identifiers, BM25 may add little.
• If latency budgets are tight, reranking 100 candidates may be too slow; reduce the candidate set, use a smaller reranker, or rerank only when the first-stage scores look uncertain.
• If your domain is highly specialized, a generic reranker may underperform, and fine-tuning one on your own labeled pairs may be needed.

Common Mistakes

• Judging the whole system by final answers only, so retrieval failures look like model failures.
• Reranking too few candidates, which caps recall before the reranker even runs.
• Mixing raw BM25 and cosine scores by simple addition without normalization.
• Feeding the reranker chunks longer than its input limit.
• Using one fixed top-k for every query type.
• Forgetting to re-evaluate after changing the embedding model, chunk size or corpus.
• Tokenizing identifiers badly for BM25, for instance splitting “ERR_4012” into pieces that no longer match. Check your analyzer.

Key Takeaways

• Most RAG failures are retrieval failures; fix the retrieval before tuning prompts.
• Dense search captures meaning but is weak on identifiers and exact constraints; BM25 is the reverse, so combining them raises recall.
• Reciprocal rank fusion is a simple, score-free way to merge ranked lists and works well as a default.
• A cross-encoder reranker reads query and passage together, giving much better ordering at the cost of running only on a short candidate list.
• Evaluate retrieval separately with recall@k and MRR on a labeled question set, and compare each stage against the simpler alternative.
• Add layers only when they pay off: small corpora and tight latency budgets may not need the full pipeline.