Back

Hybrid Search Architecture: Fusing BM25 and Vector Recall with RRF

Tengence Team ·
Vector Search Ranking Algorithm Search System
Hybrid Search Architecture: Fusing BM25 and Vector Recall with RRF

Summary: BM25 and vector recall are complements, not alternatives. What decides quality is not whether you use vectors but how two result lists become one. This article splits fusion into three dimensions and gives a path that starts from the minimum viable shape.

Key Takeaways

1. Why hybrid is mandatory

Single-path failure is structural, not a tuning problem.

Keyword retrieval fails in three ways: colloquial descriptions ("computer will not start" versus "boot failure"), cross-language and synonym gaps ("reimbursement" versus "expense claim"), and typos and abbreviations. On an inverted index, none of the query terms may match.

Vector retrieval has its own clear failures: exact identifiers (order numbers, model codes, clause numbers), extremely short queries, and proper nouns rare in the training corpus. Vectors pull semantically close items together, which is exactly how "A-1024" and "A-1025" get confused.

The two failure zones barely overlap, and that is the source of hybrid value. It is not about being more advanced; it is about covering the other path's structural blind spot.

One practical consideration: the ratio of exact queries to fuzzy queries in real traffic is stable. Any single-path system performs poorly on a fixed share of queries — and that share tends to be the high-value ones, where users look for a specific order or clause and errors cost most.

Stuck on which site-search engine to pick, or on flat results?

See search and recommendation

2. The three dimensions of fusion

Dimension Problem Common practice Trap
Candidate set How many per path, union or not Take Top-K from each, merge, deduplicate K too small, correct result never enters
Score Two incomparable scales Weighting, normalisation, RRF Adding raw scores across scales
Rank How the final order is set Fusion ranking, reranking No rerank after fusion, order still arbitrary

The candidate set is the most neglected dimension. Many implementations take the top ten from each path and end up with twenty candidates that exclude the right answer, because it ranked fifteenth on the vector path. K should be set from recall evaluation, not from how many results the page displays.

The core score problem is scale. BM25 produces an unbounded match-strength value; cosine similarity sits between 0 and 1. Adding them is meaningless. Either normalise before weighting or abandon scores and read rank only.

Rank is the source of explainability. If the fused list cannot explain why an item is first, tuning becomes guesswork. RRF's advantage is exactly this: order derives from the sum of per-path ranks, so any position is traceable.

3. Five implementation steps

Step one: size the candidate pools. Measure recall on a real query set and increase K until more does not improve recall. This step sets the ceiling for everything after.

Step two: ship RRF. Each document's score is the sum of reciprocal ranks across paths, with a smoothing constant fixed at a conventional value. RRF needs no training or labels and works as a baseline.

Step three: deduplicate and merge same-source items. When one document appears across shards or versions, merge before ranking, or the top ten may all be versions of the same page.

Step four: add reranking. A small model reorders the fused head. Its input must include the query and the passage, not scores alone, or it cannot learn semantics.

Step five: build degradation. When either path times out or fails, the other path serves the result and the response flags partial recall. Hybrid search without degradation is less available than single path.

Step three is the one most often skipped. Deduplication looks like housekeeping and directly shapes experience: users who see five duplicates in ten results conclude the search is bad, even when the ranking is fine.

4. Engineering: parallelism, timeouts, caching

Three architectural points decide stability under real traffic.

Both paths must run in parallel. Serial execution makes latency the sum of both, and tolerance for first-screen time is low. In parallel, total latency approaches the slower path.

Timeouts must be tiered. The vector path is usually slower; give it a shorter timeout and allow degradation. The keyword path is faster and serves as the backstop. Set timeouts from measured distributions, not a guessed constant.

Cache in layers. Query-level caching suits frequent repeats; passage-level caching suits the long tail. Invalidation must be tied to index updates, or users see stale results after content changes.

One more: index consistency. If the two indexes update on different cadences, a window exists where keywords find a document and vectors do not. Short windows are acceptable, but they need monitoring, or the problem gets filed as "random failure".

5. Selection criteria and risk list

Four questions decide the build.

Which query classes dominate your traffic? An internal document search with many identifiers needs strong keyword weighting; a consumer help centre needs vector strength.

Do you have labelled relevance data? Without it, learning-to-rank is an expensive hobby. Start with RRF.

What latency budget does the product allow? Hybrid plus reranking typically doubles single-path latency; the budget must be agreed before architecture.

Who maintains the index? Hybrid doubles operational surface. An unowned second index decays quietly.

Risk list: score-scale errors producing arbitrary order; K sized from display count rather than recall; duplicates crowding the head; serial execution; missing timeouts; dual-index drift; reranking added before fusion is stable; and no evaluation set, which makes every change unfalsifiable.

The order of work is the same for both retrievers: build an evaluation set from real queries first, then change one component at a time. Teams that tune fusion weights before they can measure recall on real queries end up optimising a number that never appears in production logs, and the change looks successful until traffic arrives.

FAQ

Q: Is hybrid search always better than vector-only?
On mixed traffic, yes. On a corpus of short, semantically uniform texts with no identifiers, the gain shrinks and operational cost may not justify it.

Q: What smoothing constant should RRF use?
The conventional value of 60 is a reasonable default. Tune only if you have an evaluation set that shows a consistent improvement.

Q: Do we need a reranker at all?
Only if the top of the fused list still looks wrong. Reranking cannot rescue a candidate set that never contained the answer.

Q: How do we measure whether fusion helped?
Recall at K, NDCG on a labelled set, and zero-result rate on real queries. Track all three; a single aggregate hides regressions.

Q: Can BM25 and vectors share one index?
Some engines support both in one index, which simplifies consistency. If they are separate, a single write path and a lag monitor are mandatory.

Related Reading

Data Sources

  1. k-nearest neighbor search — Elastic documentation — dense vector retrieval and how it combines with lexical scoring.
  2. What is Azure AI Search — Microsoft Learn — hybrid retrieval and semantic ranking in production search services.
  3. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020 — why retrieval quality bounds downstream answer quality.
← Previous: On-Premise AI Search in 2026: Architecture and Capacity Planning Next: Experiment Analysis Framework: Metrics, Checks and Decisions →
Consult
QR code

WeCom