Hybrid Retrieval for RAG: BM25, Embeddings, Fusion and Reranking

What is hybrid retrieval?

Retrieval-augmented generation (RAG) answers a question by first finding the passages that matter, then asking a model to write from them. Hybrid retrieval is the retrieval stage done properly: it runs keyword search (BM25) and dense search (embeddings) together, fuses the two ranked lists, and reranks a shortlist with a more precise model. It exists because neither retriever is enough on its own.

The examples below come from a freight-forwarding company, where shipping lanes carry codes such as INNSA (Nhava Sheva) and NLRTM (Rotterdam) and analysts ask questions about invoices and rate cards.

1. 1QuestionThe user asks, in their own words
2. 2Two searchesBM25 matches the exact words; embeddings match the meaning
3. 3FuseReciprocal rank fusion merges the two ranked lists by rank
4. 4RerankA slower, sharper model re-scores only the shortlist
5. 5Answer from the best fewPass the fewest chunks that contain the answer

Two retrievers, two blind spots

Neither is enough on its own. Production systems use both and fuse the results.

The whole pipeline, in order:

1. The analyst's question arrives, with optional rewriting (codes and synonyms).
2. BM25 returns its top 50 chunks and dense search returns its top 50.
3. Reciprocal rank fusion merges the two lists into one.
4. A cross-encoder reranks the fused shortlist.
5. The top k chunks that fit the token budget go to the model.

How BM25 scores (the intuition)

BM25 rewards documents that contain the query's terms, more so when the term is rare across the collection (an invoice number appears in one document; "the" appears in all of them), with diminishing returns for repeating a term and a penalty for very long documents. You rarely implement it yourself, but you must know what it is good at: exact, rare terms.

Reciprocal rank fusion, step by step

BM25 scores and cosine similarities live on different scales, so you cannot add them. Reciprocal rank fusion (RRF) uses ranks instead:

score(item) = sum over each list of 1 / (k + rank), with ranks starting at 1, and k usually 60.

Worked example for "What charges are on INV-77812?":

The invoice reaches the top 3, though not first. Fusion fixed recall; the order is the reranker's job.

def rrf(rankings, k=60):
    scores, first_seen = {}, []
    for ranking in rankings:
        for rank, item in enumerate(ranking, start=1):      # ranks start at 1
            if item not in scores:
                first_seen.append(item)
            scores[item] = scores.get(item, 0.0) + 1.0 / (k + rank)
    return sorted(first_seen, key=lambda item: -scores[item])   # stable sort keeps first-seen order on ties

Rerankers: precise, slow, shortlist only

A cross-encoder reads the question and one chunk together and scores how well the chunk answers it. It is far more precise than comparing two independent embeddings, and far slower, so it only scores a shortlist (typically the top 20 to 50 fused results). In the example above it would move the invoice (c7) from third place to first.

The key consequence: a reranker improves order (MRR, precision), never recall. A chunk that was never retrieved cannot be rescued. If the right chunk is not in the shortlist, fix retrieval, not the reranker.

Query rewriting

Treat every rewrite as an experiment: keep it only if a benchmark metric improves.

Measure before you tune

Build a benchmark first: real analyst questions, each labelled with its relevant chunks. Then measure every change.

def recall_at_k(ranked, relevant, k):
    return len(set(ranked[:k]) & relevant) / len(relevant) if relevant else 0.0

def reciprocal_rank(ranked, relevant):
    return next((1 / rank for rank, item in enumerate(ranked, start=1) if item in relevant), 0.0)

Context engineering: the fewest chunks that contain the answer

Sending more chunks costs tokens on every request, adds latency, and can distract the model with near-misses. After good retrieval and reranking, the answer's evidence is usually in the top 3 to 5.

Put hard constraints in filters before ranking (lane, validity date), not in the hope that ranking gets them right.

A checklist for your own system

• A labelled benchmark exists, and every change is measured on it.
• BM25 and dense retrieval are fused with RRF, and the value of k is recorded in a design decision.
• A cross-encoder reranks a shortlist; recall problems are fixed in retrieval, not the reranker.
• Metadata filters enforce lane and validity before ranking.
• The context holds the fewest chunks that keep context recall on target, within the token budget.

Retrieval is one skill of the forward deployed engineer. For the role as a whole, read what a forward deployed engineer does.