Hybrid Search and Re-Ranking: Beyond Pure Vector Similarity

Why pure vector search is not enough

A vector database demo answers three plausible questions flawlessly and a team concludes the retrieval problem is solved. Then the pipeline meets real enterprise traffic, and a class of queries starts failing in a way that looks like the model is broken but is not. A user searches for error code ORA-01555, part number MX-4470-B, or the acronym DORA, and the dense retriever returns passages that are thematically adjacent and factually useless. The exact token that mattered never drove the match.

The reason is structural, not a tuning miss. A dense embedding compresses a passage into a fixed-size vector that captures meaning, and meaning is exactly what smooths over surface form. Two strings that differ by one critical character sit close in embedding space; a rare product code and a common one look nearly identical to a model that has seen neither often. Dense retrieval is therefore strongest on paraphrase and synonymy and weakest on precisely the tokens that are rare, out-of-vocabulary, or semantically load-bearing because they are identifiers rather than concepts.

Negation is the other quiet failure. "Contracts that do not auto-renew" and "contracts that auto-renew" embed close together because they share almost all their words and most of their meaning; the one word that inverts the intent barely moves the vector. Lexical search has the mirror weakness: it matches strings, so a query for "employee termination policy" sails past a document titled "guidance on separating staff" because not one content word overlaps. Each method is blind exactly where the other sees.

Lexical search, still essential

Lexical retrieval predates the transformer era and, for a large slice of enterprise queries, has never been beaten. The workhorse is BM25, the standard sparse ranking function. It scores a document against a query on two intuitions that have held up for decades: a document that contains the query terms more often is more relevant, with sharply diminishing returns, and a term that is rare across the whole corpus is more discriminating than a common one. Match on a word that appears in every document and you have learned nothing; match on a word that appears in three documents and you have almost certainly found one of them.

That inverse-document-frequency weighting is exactly why lexical search dominates on the tokens dense retrieval fumbles. A part number appears in a handful of documents, so its rarity makes it a near-perfect selector; BM25 ranks the passage that contains it at the top without any training, embedding, or model at all. The same holds for error codes, statute references, ticker symbols, person names and internal acronyms: the rarer and more exact the term, the more decisively lexical search wins.

The operational virtues are underrated in the rush to vectors. Sparse retrieval is cheap to run and cheap to update, since indexing a new document is an inverted-index write rather than a GPU embedding pass. It is also interpretable: you can see precisely which query terms matched which passage and why one outranked another, which matters when a compliance owner asks why a given document surfaced. Dense retrieval offers no such account of itself.

Fusing lexical and dense results

Hybrid search is less clever than it sounds: run both retrievers over the same query, then merge their result lists into one ranking. The whole difficulty lives in the merge, because the two retrievers speak incompatible languages. BM25 emits unbounded relevance scores whose scale shifts with query length and corpus statistics; a cosine similarity from the vector index lives in a fixed, narrow band. Adding or comparing these numbers directly is meaningless, and a naive sum lets whichever score happens to run larger silently dominate the result.

Reciprocal Rank Fusion sidesteps the problem by throwing the scores away and keeping only the ranks. Each document gets, from each retriever, a contribution of 1/(k+rank), where rank is its position in that retriever's list and k is a small constant (often around 60) that damps the influence of the very top positions. Sum a document's contributions across both lists and sort. A passage ranked first by lexical search and eighth by dense search scores well on both counts and rises; a passage that only one retriever liked still gets a fair hearing.

 QUERY
   |
   +--> BM25        --> [d3, d7, d1, d9, ...]   ranked list
   |
   +--> vector KNN  --> [d7, d2, d3, d5, ...]   ranked list
             |
             v
   RRF: score(d) = sum over lists of 1/(k + rank)
             |
             v
   d7: 1/(k+1) + 1/(k+2)   <- top of both, wins
   d3: 1/(k+3) + 1/(k+1)   <- strong on both
   ...
   FUSED RANKING --> top-k passes to re-ranker
            

The reason RRF is the pragmatic default is that it requires no tuning of score scales and degrades gracefully when one retriever is noisy. Weighted score fusion, where you normalize each retriever's scores and combine them with a learned or hand-set weight, can edge out RRF when you have invested in careful calibration and a labeled tuning set. Absent that investment, its normalization is fragile and drifts as the corpus changes. Start with RRF; reach for weighted fusion only when a golden set proves it earns the extra maintenance.

Re-ranking with cross-encoders

Fusion produces a good wide net, not a precise top of the list. To understand why, separate the two model architectures at play. The retriever is a bi-encoder: it embeds the query and every document independently, so document vectors are computed once at ingestion and a query becomes a fast nearest-neighbor lookup. That independence is what makes retrieval scale to millions of passages, and it is also its ceiling, because the model never sees query and document together and cannot reason about how they actually relate.

A cross-encoder re-ranker removes that constraint. It takes the query and one candidate document as a single joined input and outputs a relevance score, letting every query token attend to every document token. The judgment is far sharper: it can tell that a passage mentioning your query terms is actually about something else, or that a paraphrase with no shared words is a precise answer. The cost is that there is no precomputation. Each query-document pair is a full forward pass, so scoring is orders of magnitude more expensive per candidate than a vector lookup and adds real latency.

The resolution is the pattern that defines a modern retrieval stack: retrieve wide, then rerank narrow. Let hybrid search cast a wide, cheap net and return perhaps the top 50 to 100 candidates, optimizing for recall, getting the right passage somewhere in the pool. Then spend the cross-encoder only on that small set to reorder it for precision, and pass the top handful to the generator. You pay the expensive model on dozens of candidates, not millions.

Metadata and access filtering

Retrieval in an enterprise is never a pure relevance problem. The most relevant passage in the corpus is worthless, or actively dangerous, if the user asking is not entitled to see the document it came from. Access control cannot be delegated to the prompt or the generator; a model instructed to ignore documents outside a user's permissions is one clever question away from leaking them. Entitlements have to be enforced as a hard filter in the retrieval query itself, using the ACL or security tag that every chunk should carry as metadata (see the chunking deep-dive for why that metadata belongs on the chunk).

Beyond entitlements, metadata filtering is how you enforce the non-semantic constraints a query implies. Freshness is the common one: a question about current policy should not surface a superseded version, so you filter or down-weight by effective date. Source, document type, language, region and department round out the usual set. Filters run either before retrieval (restrict the candidate space, then search) or after (retrieve, then drop disallowed hits); pre-filtering is safer for entitlements because it never materializes forbidden results, while post-filtering can be simpler when the filter is broad.

The trap is evaluating retrieval with filters off and shipping with them on. A filter changes the candidate pool, and therefore changes recall: a strict date window or a narrow ACL can starve a query that looked healthy in an unfiltered test. If pre-filtering leaves too few candidates for the re-ranker to work with, top-k precision quietly collapses even though the ranker is unchanged. Measure the stack with production filters applied, per user role if entitlements vary, so the numbers you trust are the numbers users experience.

Tuning and measuring the stack

A hybrid pipeline with re-ranking exposes a handful of parameters that interact, and the only honest way to set them is to measure. The knobs that matter: how many candidates each retriever returns, the RRF constant (or fusion weights if you went that route), how wide the fused candidate pool is before reranking, and how deep the re-ranker reorders before the top few reach the generator. Guessing these from intuition is how teams end up with a re-ranker that never sees the right passage because retrieval was too shallow.

The instrument is the same golden set that governs every other retrieval decision: 100 to 300 real user questions, each labeled with the passages that answer it, drawn from logs where possible. Trend two numbers as you sweep. Recall at the retrieval stage tells you whether the right passage entered the candidate pool at all, which is the re-ranker's raw material; precision at the final k tells you whether it reached the top after reranking. A re-ranker cannot fix a recall failure, so read recall first: if the passage never made the pool, widen retrieval before touching the ranker. The mechanics slot into the same eval discipline as the rest of the pipeline, covered in Evaluating RAG, and the RAG Explorer in the Lab lets you watch the ranking shift as you move each knob.

The table is deliberately directional rather than numeric: the exact lifts depend on your corpus, query mix and models, and quoting someone else's percentages as if they were yours is how pipelines get shipped on false confidence. Run the sweep, record the real deltas, and gate changes on them.

The default retrieval stack

Strip away the debate and a clear default emerges for enterprise RAG: hybrid retrieval (dense plus BM25, fused with RRF) feeding a cross-encoder re-ranker over a wide candidate pool, with metadata and access filters applied throughout. This is not the most sophisticated architecture available; it is the one that survives contact with real query traffic, mixed content, and a corpus full of identifiers, acronyms and negations that pure vector search handles badly.

 query --> [ metadata + ACL filter ]
             |
      +------+------+
      |             |
    BM25         vector KNN        (retrieve wide, high recall)
      |             |
      +---> RRF fusion ---> ~50-100 candidates
                   |
            cross-encoder re-rank   (rerank narrow, high precision)
                   |
            top 3-8 passages --> generator
            

Build it in that order, because each stage earns its place before the next is added. Hybrid search is nearly free and closes the exact-term blind spot immediately. The re-ranker is the next increment and typically the largest single precision gain, for a bounded cost you can cap by fixing the candidate count. Only then, if evaluation shows queries still failing for reasons of structure rather than ranking, does a heavier tool earn consideration.

That heavier tool is usually graph retrieval, and it is a targeted addition, not a default. Multi-hop questions that require connecting facts across several documents ("which suppliers are two steps removed from this sanctioned entity") are a different shape of problem that ranking passages does not solve; that is where GraphRAG pays off. For the overwhelming majority of enterprise questions, which ask for one well-scoped fact from one right passage, hybrid plus a re-ranker is the stack to reach for first and the one to beat before adding anything more exotic.