How to Optimize Search Relevance Ranking? A Two-Stage Recall + Ranking Method

Summary: Search relevance problems must first be diagnosed as either a recall failure or a ranking failure: when recall is insufficient, expand the candidate sources and recall width; when ranking is off, add features and models. The two troubleshooting paths are nearly orthogonal, and mixing them in one change usually fails.

When users complain that "the search results are irrelevant," the problem may lie in two completely different stages: the target document never entered the candidate set (recall failure), or it entered the candidate set but ranked 40th (ranking failure). Tuning ranking weights is useless for the former, while adding recall paths for the latter only injects more noise into the tail. This article breaks relevance down by the "recall–ranking" two-stage model: a decision table for the two problem types, the funnel volume of multi-path recall, the division of labor between coarse ranking and fine ranking, fine-ranking features and the three families of Learning to Rank, fixes for four types of ranking distortion, plus offline evaluation and a three-stage rollout path.

Key Takeaways

Relevance problems split into two types first: cannot recall vs. cannot rank up; the criterion is whether the target document appears in the candidate set.
The recall stage mainly watches Recall@K and the candidate volume at each funnel layer; the value of multi-path recall is covering different matching logics.
Coarse ranking narrows candidates from the tens of thousands to the hundreds; fine ranking decides the order from hundreds to tens; their latency budgets are completely different.
Fine-ranking features fall into four classes: text relevance, user behavior, document attributes, and context; missing any one produces predictable distortion.
Pointwise, pairwise, and listwise in Learning to Rank trade off differently on labeling cost and optimization objective.
An offline metric gain does not equal an online gain; significance testing and segmentation by query type are mandatory.

1. Distinguish the Two Problem Types First: Missing from Recall, or Failing to Rank Up

The first step in optimizing relevance is not tuning parameters but locating the problem. The same "search results are irrelevant" complaint may come from two orthogonal stages: the target document is not in the candidate set (insufficient recall), or the document is in the candidate set but ranked near the bottom (improper ranking).

The diagnostic method is offline replay: take a batch of bad-case queries (zero results, no clicks, overly deep paging), manually label the documents that should have appeared, then rerun the pipeline to see whether they entered the recall candidate set. The criterion is whether the target document entered the candidates, not where it finally ranked. Counting a few dozen cases is enough to set priorities: if it did not enter the candidates, do recall first; if it entered but ranked beyond dozens of positions, do ranking first.

Dimension	Insufficient recall	Improper ranking
Diagnostic signal	Target document not in the recall candidate set	Target document in the candidate set, low after fine ranking
Typical symptom	Zero results, very few results, cannot find even at the bottom	Many results, wrong top pages, found only by deep paging
Main metric	Recall@K, zero-result rate	NDCG@10, MRR, top-position hit rate
Effective means	Add recall paths, widen recall, improve query understanding	Add features, deploy fine-ranking model, correct position bias
Common ineffective action	Tune ranking weights, add fine-ranking features	Keep adding recall paths, enlarge candidate set

2. Recall Stage: Multi-Path Recall and Funnel Volume

The goal of recall is not "accurate ranking" but to pull as many potentially relevant documents into the candidate set as possible at an acceptable cost. A single recall path inevitably has blind spots; the mainstream approach runs four paths in parallel.

Literal recall is based on the inverted index and term-frequency statistics (such as BM25), strong at exact matches of model numbers and codes; vector recall maps queries and documents into the same vector space to bridge the vocabulary gap of inconsistent wording, and the boundary and fusion of the two are covered in What Is Semantic Search? A Full Comparison of Vector Retrieval vs. Keyword Retrieval; structured filtering handles hard constraints such as category, price, and inventory, and should be a filter condition rather than a scoring signal; popularity and personalized fallback ensure there is still content to show for very short queries.

The multi-path results need fusion. Elasticsearch officially defines Reciprocal Rank Fusion (RRF) as "a method that combines multiple result sets with different relevance metrics into a single result set," with the advantage of requiring no parameter tuning and no need for the path metrics to be on a comparable scale.

The funnel volume is: full corpus → after filtering, several thousand to tens of thousands → after coarse ranking, several hundred → after fine ranking, dozens → display ten. The candidate volume at each layer is the ceiling for the next. Recall rate is measured by Recall@K. The difficulty is the ground truth: the full set of relevant documents relies either on a labeled set or on click-log inference, and the latter is inherently incomplete.

3. Ranking Stage: How Coarse Ranking and Fine Ranking Divide the Work

Ranking is split into two stages because the candidate volume and latency budget are not on the same scale. Coarse ranking faces thousands to tens of thousands of candidates and can only use lightweight scoring computable inside the inverted or vector store (BM25 score, vector similarity, static quality score) combined with linear fusion, with a latency budget of single-digit milliseconds; its only task is to filter candidates from the tens of thousands down to the hundreds without prematurely eliminating good documents.

Fine ranking faces hundreds of candidates and can use full features and complex models (GBDT, LambdaMART, neural networks), with the latency budget relaxed to tens of milliseconds, implemented via rescore. Elasticsearch's official documentation positions Learning to Rank exactly this way: the model usually serves as a second-stage reranker to improve the relevance of the first-stage recall results.

After fine ranking there is another layer of reranking, the business-rule layer: pinning, blocking, deduplication, diversification, compliance filtering. It is not a fallback but the source of business controllability. The three-layer architecture is decomposed in What Is a Recommendation System? Recall, Ranking, and Reranking Three-Layer Architecture; search and recommendation are isomorphic at this layer.

4. Fine-Ranking Features and the Trade-Offs of Learning to Rank

Fine-ranking features are the carrier of relevance; it is recommended to build all four classes: text relevance (BM25 score, field hit, query term coverage, term adjacency, vector similarity); user behavior (click-through rate, add-to-cart rate, conversion rate, must be smoothed by exposure volume, otherwise noise drowns the signal); document attributes (timeliness, quality score, rating, sales, inventory); context (device, time slot, query type, user history).

With features in place, the next question is what objective to combine them with, i.e., the method selection of Learning to Rank.

Method	Label form	Optimization objective	Use and cost
Pointwise	Relevance grade of a single document	Single-point prediction error	Simplest labeling; but objective is disconnected from ranking position
Pairwise	Partial order of document pairs	Number of inverted pairs	Labels can be built from click pairs; ignores the whole list, tends to cluster
Listwise	Order of the entire list	Directly optimize metrics such as NDCG	Closest to online objective; highest labeling and computation cost

For selection: when labeling relies on humans and data volume is small, start from pointwise; after stable click logs, move to pairwise; only adopt listwise when you truly need to optimize list-level objectives (diversity, overall revenue). It should be noted that Airbnb pointed out in public research that the assumption implicitly made by common LTR frameworks—"the value of one result can be determined independently of other items in the list"—does not hold; this is precisely the structural reason why pointwise-only systems easily end up with "the top ten look all the same."

5. Four Common Ranking Distortions and Their Fixes

Relevance drowned by sales. Behavioral features have far larger scale and coverage than text features, and the model quickly learns "rank high sales first." The fix is to set text relevance as a hard threshold: documents with text score below the threshold cannot enter the head, then use quantile calibration and bucket normalization to compress behavioral features to a comparable scale.

Long-tail query collapse. Training samples are dominated by head queries, and long-tail query ranking is nearly random. The fix is stratified sampling or weighting by query frequency to preserve the proportion of long-tail samples in the training set; at the same time, raise text feature weight and lower behavioral feature weight for long-tail queries.

New content cannot rank up. No exposure means no clicks, forming a dead loop. There are three fixes: ensure index real-time so new content is recallable within minutes; use exposure-independent features (content quality score, attribute completeness) for cold-start scoring; reserve a fixed proportion of exploration traffic to accumulate data. The causes and reductions of the zero-result rate are specifically decomposed in How to Reduce the Zero-Result Rate of Site Search? 8 Actionable Strategies.

Position bias. Results higher up are more likely to be clicked; treating clicks directly as relevance labels freezes the historical ranking. The unbiased learning-to-rank method proposed by Microsoft Research at SIGIR 2018 uses propensity estimation (IPS) for correction: weight clicks by "the probability that a result at that position is observed by the user," learning an unbiased ranking model from biased logs. A lighter approach is to maintain a small amount of randomized or position-swapped exploration traffic.

6. Evaluation and Rollout: Offline Metrics and a Three-Stage Path

Offline metrics solve "can we iterate quickly," and online A/B solves "is it really useful." MRR suits navigational and single-answer queries; MAP examines the overall quality of multiple relevant results; NDCG@K considers both multi-level relevance and position discount, and is the default choice for multi-result scenarios. Elasticsearch's ranking evaluation interface supports metrics such as precision, recall@k, MRR, and DCG, and can directly evaluate a labeled set.

Build the labeled test set like this: stratified sampling from real query logs, with head, torso, and long-tail each at a fixed proportion; label the relevance grade of 20–50 candidates per query; two people independently label and check consistency; periodically re-label after content updates. Behavioral signals can serve as weak labels but carry position bias.

Segmented evaluation is mandatory, by query type, whether there are results, and category. Overall metrics are averaged out by head queries; the most common failure is "overall went up, long tail collapsed" going unnoticed. In addition, the advantage of LTR is not always stable: statistical tests on multiple public datasets show that many LTR algorithms produce no significant difference relative to "the single best feature in the dataset," and most baselines are statistically tied, so offline evaluation must include significance testing.

Rollout is in three stages. Stage 1 builds observation and baseline: complete query-term, exposure, click, no-result, and query-rewrite instrumentation; build a labeled set and run segmented baselines; watch the zero-result rate and top-position click rate. Stage 2 adds recall and coarse ranking: connect multi-path recall and RRF fusion, do query understanding (correction, synonym normalization, intent recognition), strip filter conditions from scoring, and watch Recall@K and long-tail candidate coverage. Stage 3 deploys fine ranking and the rule layer: build the four-class features, train an LTR model, configure business rules while keeping exploration traffic, and watch the segmented lift of NDCG@10 and online A/B significance. At this stage search and recommendation can share the ranking base; the collaboration is covered in Synergy of Recommendation Systems and Search: An Integrated Growth Solution.

FAQ

Q: How can I quickly tell whether poor search relevance is a recall problem or a ranking problem?

A: Do offline replay: take a batch of bad-case queries, manually label the documents that should have appeared, and rerun the pipeline to see whether they entered the recall candidate set. Not entering the candidates is a recall problem; entering the candidates but dropping to dozens of positions after fine ranking is a ranking problem.

Q: Is more recall paths always better?

A: No. Each added path costs indexing, latency, and fusion; it is only worth adding when it covers an existing blind spot. The criterion is incremental recall rate: if the proportion of relevant documents the new path brings that other paths cannot recall is too low, abandon it.

Q: Can I do Learning to Rank without labeled data?

A: Yes, but lower your expectations. First use click logs to construct preference pairs and train a pairwise model, while building a small-scale manual labeled set for validation. Pure behavioral labels carry position bias; you must reserve exploration traffic or use inverse-propensity weighting for correction, otherwise the model only reinforces historical ranking.

Q: Why did offline NDCG improve but online conversion rate did not change?

A: Three common reasons: no significance testing was done, so the gain falls in the noise range; overall metrics are averaged out by head queries, while the long tail actually worsened; the relevance labeled offline is not the same as purchase intent, so the labeling caliber needs checking.

Q: What if newly listed content always fails to rank up?

A: First confirm whether it can be recalled: when index real-time is insufficient, new content is not in the candidate set, which is a recall problem. If it can be recalled but still cannot rank up, it is a cold start caused by missing behavioral features, and you need to score with features such as content quality score and attribute completeness.

Q: After deploying a fine-ranking model, do I still need to keep manual rules?

A: Yes. Pinning, blocking, diversification, and compliance filtering are the source of business controllability, which the model cannot guarantee. The division is that the model handles relevance ranking and the rule layer handles business constraints and diversity, overriding results only where necessary.

About Tengence

Tengence Search addresses the relevance-tuning scenarios described in this article, providing multi-path recall, reciprocal rank fusion, Learning to Rank fine ranking, and manual rule intervention, with offline evaluation and online A/B linkage. Tengence (TENGENCE) is the enterprise growth engine in the AI era, focused on public-traffic acquisition and private-domain conversion; its core products are Tengence GEO (dual-engine GEO + SEO) and Tengence Search.

Data Sources

arXiv "Unbiased Learning to Rank with Unbiased Propensity Estimation" (Microsoft Research, 2018): supports the claim that position bias can be corrected with propensity estimation (IPS).
arXiv "Is Learning to Rank Worth It? A Statistical Analysis of Learning to Rank Methods" (2013): supports the claim that offline gains must undergo significance testing and most baselines tie with LTR statistically.
arXiv "Learning To Rank Diversely At Airbnb" (Airbnb, 2022): supports the claim that the value of one result cannot be determined independently of the result list.
Elasticsearch Official Guide "Learning To Rank (LTR)": supports the two-stage claim that "the LTR model usually serves as a second-stage reranker to improve the relevance of the first-stage recall results."
Elasticsearch Official Guide "Reciprocal rank fusion": supports the claim that "RRF merges multi-path result sets with different relevance metrics into a single result set without parameter tuning."
Elasticsearch Official Guide "Evaluate ranked search results": supports the claim that "the ranking evaluation interface supports offline metrics such as precision, recall@k, MRR, and DCG."