What Is a Recommendation System? Retrieval, Ranking, and Re-ranking Explained

A recommendation system is a decision system that selects the small set of content a user is most likely to consume from a massive candidate pool. The mainstream architecture in industry connects three layers: retrieval, ranking, and re-ranking. It first narrows millions of candidates to thousands, calculates precise scores for hundreds, and then applies business rules to determine the final display positions.

Definition of a Recommendation System: From People Finding Information to Information Finding People

A recommendation system, or recommender system, is a type of information filtering system. It combines user behavior, item attributes, and contextual signals to predict a user's preference for content they have not yet consumed, then sorts the output by predicted preference. According to Google's official machine learning course, recommendation models predict preferences by analyzing similarities among items and users' historical interactions. Two common forms are personalized home-page recommendations and related-item recommendations. See Recommendation Systems Overview.

The boundary between recommendation and search is whether the user provides an explicit query. Search is people finding information: users express intent with keywords, and the system performs relevance matching. Recommendation is information finding people: without a query, the system must infer intent from historical behavior. Both share relevance calculations, index structures, and ranking models at the underlying level, so engineering teams often place them in the same technology stack. See Integrated E-commerce Search and Recommendation: From Connected Data to Collaborative Optimization.

Three types of input determine the upper limit of recommendation quality:

Behavioral signals: Impressions, clicks, add-to-cart actions, favorites, purchases, viewing time, and bounces. These are the densest signals and the closest to actual preferences.
Item attributes: Category, price band, title, and image-text vectors. These determine what the system can still recommend during cold start.
Context: Time of day, device, location, and preceding behavior within the session. These explain why the same person sees different results on the home page and product detail pages.

Why Three Layers Are Necessary: The Conflict Between Scale and Latency

In theory, the optimal approach is to score every candidate in the full catalog with the most sophisticated model. In practice, candidate scale and response latency impose hard constraints. A classic paper published in IEEE Internet Computing in 2003, written by authors from a global e-commerce company, provided figures that remain representative: large retailers often serve tens of millions of customers and millions of products, while recommendation results must return within 0.5 seconds without compromising quality. See the Item-to-Item Collaborative Filtering paper, 2003.

Within a 0.5-second budget, precise scoring can usually handle only thousands to tens of thousands of candidates, while the candidate pool may contain millions. This is the sole reason for the three-layer architecture: use inexpensive algorithms to narrow the range quickly and reserve costly computing for a small set of high-value candidates.

First Layer: Retrieval, or Candidate Generation

The goal of retrieval is not precision but breadth. Low-cost methods select hundreds or thousands of potentially relevant items from millions of candidates. It is better to retrieve extra items than to miss what the user truly wants.

Mainstream retrieval methods fall into four categories:

Collaborative filtering: Includes user-based and item-based methods. Item-to-item collaborative filtering is the classic engineering solution. It calculates an item-similarity table offline, then retrieves similar items online based on the user's recent behavior. Its complexity is independent of the number of users, making it naturally suitable for a large catalog.
Vector retrieval, using two-tower models or embeddings: Encode users and items separately as vectors, then retrieve Top-K results online through approximate nearest neighbor search, or ANN. This approach solves collaborative filtering's limited ability to generalize to long-tail and new items and is now mainstream.
Matrix factorization and graph models: ALS and BPR process implicit feedback, while graph neural networks model higher-order relationships.
Rules and fallback strategies: Popularity charts, regional bestsellers, new-item pools, and editorial allowlists maintain coverage and control.

Industrial systems typically run and merge 5–20 retrieval channels in parallel. Google's course explicitly notes that candidates can come from different sources, including user features, popular items, and social graphs, before being passed to a separate model for unified scoring. See Retrieval.

Retrieval is evaluated with Recall@K, which measures how many consumed items entered the candidate set, along with coverage and response latency.

Second Layer: Ranking, or Scoring

The ranking layer receives thousands of candidates and uses sophisticated models to estimate the target probability for each candidate, such as click-through rate, or pCTR, conversion rate, or pCVR, viewing time, and GMV. It then weights and sorts candidates according to business goals and outputs hundreds of results.

There are three key engineering considerations:

Feature system: User-side features such as profile tags and long-term or short-term interests, item-side features such as price, sales, and quality scores, contextual features such as time, device, and session, and cross-features among all three. Tag-system design directly determines the model's performance ceiling. See the User Tagging System Design Guide.
Unified scoring model: Google's course emphasizes that a single model should score candidates from different retrieval sources. This makes scores more comparable than directly comparing the native scores of each source and also makes context easier to incorporate. See Scoring.
Position-bias correction: Display position contaminates training samples because items shown near the top are naturally more likely to receive clicks. The correct approach is to pursue position-independent ranking, such as scoring every candidate as though it occupied the first position.

The typical model evolution is logistic regression → gradient-boosted decision trees plus logistic regression → deep models such as DeepFM and DIN → multi-objective learning that optimizes clicks and conversions simultaneously. Evaluation metrics include AUC, GAUC, and NDCG.

Third Layer: Re-ranking

The re-ranking layer optimizes the list as a whole rather than each item independently. It receives hundreds of candidates and creates the final display sequence under business constraints.

Common methods include:

Hard filtering: Remove already purchased duplicates, items with inventory at 0, noncompliant categories, regionally unavailable items, and content that fails safety review.
Diversity controls: Limit consecutive appearances from the same category or price band so one category does not fill the home page.
Freshness and exploration: Give exploration traffic to new and low-exposure items to prevent the system from entering a feedback loop in which recommendations become progressively narrower.
Score transformation: Google's course recommends transforming or filtering scores according to criteria such as video age and clickbait detection, then maintaining freshness through regular retraining and new data. See Re-ranking.
Fairness and compliance: Monitor impression distributions across audience groups to prevent the model from amplifying existing bias.

Post-re-ranking metrics apply to the list: diversity, novelty, average time spent per user, return rate, and the share of exposure allocated to long-tail items.

Three-Layer Architecture Comparison

Dimension	Retrieval	Ranking	Re-ranking
Input scale	Millions of full-catalog candidates	Thousands of candidates	Hundreds of candidates
Output scale	Thousands	Hundreds	Display positions, 10–50
Core objective	Maximize coverage without misses	Estimate accurately	Maximize total list value
Latency budget	10–30 ms	30–80 ms	5–20 ms
Typical methods	itemCF, two-tower vector ANN, popularity rules	LR / GBDT+LR / DeepFM / multi-objective models	Rule-based diversification, score transformation, DPP, sequence re-ranking
Core metrics	Recall@K and coverage	AUC, GAUC, NDCG	Diversity, time spent, return rate
Common failure	Highly overlapping retrieval channels collapse coverage	Uncorrected position bias creates increasingly biased recommendations	Optimizing only item-level CTR creates a homogeneous list

The hard constraint on end-to-end latency comes from the same 2003 paper: in scenarios with tens of millions of customers and millions of products, recommendation results must return within 0.5 seconds.

Case Study: Three-Layer Upgrade for a Cross-Border Home-Goods Retailer

Background: A cross-border home-goods e-commerce platform carried approximately 1200000 SKUs, served about 3000000 monthly active users, and had recommendation placements on both the home page and product detail pages.

Problem: Recommendation CTR had remained at 1.8%, and recommendation-driven GMV accounted for 12% of the total. More than 6 items from the same category frequently appeared in the top 10 positions on the home page. P95 response time was 480 ms, close to timeout.

Solution: Rebuild the system as a three-layer architecture. Retrieval merged three channels: itemCF, two-tower vectors, and regional bestsellers. Ranking used a deep model to estimate clicks and conversions simultaneously. Re-ranking added category diversification, price-band layering, and inventory filtering.

Phased actions: Weeks 1–2 completed instrumentation and sample cleaning. Weeks 3–5 launched the three retrieval channels and validated coverage. Weeks 6–8 trained the ranking model and integrated position-bias correction. Weeks 9–10 launched re-ranking strategies. Weeks 11–12 ran A/B experiments and scaled traffic.

Results after 12 weeks: Recommendation CTR increased from 1.8% to 3.1%, a +72% gain. Recommendation-driven GMV rose from 12% to 19%. The average number of categories in the top 10 home-page positions increased from 2.1 to 3.4. P95 response time fell from 480 ms to 310 ms.

Return on investment: A team of 3 worked for 12 weeks. The primary added costs were the vector retrieval database and model-training compute. Incremental returns covered the investment in month 2 after launch.

Implementation Checklist and Common Mistakes

Mistake one: Replacing the model with a larger one immediately. When retrieval coverage is insufficient, a deeper ranking model provides very little benefit. Measure Recall@K first, then decide where to invest.

Mistake two: Monitoring only CTR. Optimizing item-level CTR alone makes lists homogeneous and starves the long tail. Diversity and novelty must be monitored at the same time.

Mistake three: Ignoring position bias. Without correction, the model learns that items placed near the top are preferred by users. Recommendations then become increasingly biased after launch.

Implementation checklist:

Establish a Recall@K baseline and confirm that overlap among retrieval channels is below 40%
Correct position bias in ranking-model training samples
Ensure the re-ranking layer supports category and price-band diversification plus hard filtering for inventory and compliance
Keep end-to-end P95 latency within 500 ms
Maintain separate cold-start strategies for new users and new items
Include both item-level metrics, CTR and CVR, and list-level metrics, diversity and long-tail exposure, on the dashboard

FAQ

Q: What is the fundamental difference between recommendation systems and search engines?

A: The key difference is whether an explicit query exists. In search, users express intent with keywords, the system performs relevance matching, and success means that results match the query. Recommendation has no query, so the system must infer intent from behavior and context; success means that users consume the suggested content. Both share underlying indexing and ranking technology, so they are often built as an integrated system.

Q: Can retrieval and ranking be combined into one layer?

A: They can be combined when the candidate set is small, below tens of thousands, by using one model to score the entire catalog. This is not practical with millions of candidates and a 0.5-second latency budget. The inference cost of a sophisticated model limits it to thousands of candidates, so retrieval must narrow the range first. This engineering constraint was already established in the classic 2003 paper.

Q: Does a small team with only tens of thousands of SKUs need a three-layer architecture?

A: The team should retain the layered concept but can simplify the implementation. Retrieval can start with itemCF and a popularity chart. Ranking can use gradient-boosted decision trees without a deep model. Re-ranking can begin with rule-based diversification. The true boundary between layers should be determined by candidate scale divided by the latency budget, not by team size.

Q: Will diversification during re-ranking hurt click-through rate?

A: Item-level CTR usually declines slightly in the short term because diversification reduces consecutive exposure for the most popular items. List-level metrics, including time spent per user, browsing depth, long-tail exposure, and return-visit rate, often improve, producing higher long-term GMV. Treat re-ranking as a multi-objective trade-off and determine diversification strength through A/B experiments instead of applying it to all traffic immediately.

Q: Which layer should solve cold start?

A: All three layers have distinct roles. Retrieval uses content similarity and rule-based pools, such as new-item pools and regional bestsellers, to give new items exposure. Ranking falls back to item attributes rather than behavioral features so a lack of historical data does not force low scores. Re-ranking reserves a fixed exploration quota for new items. The same principle applies to new users: start with coarse recommendations based on context and demographic attributes, then converge quickly as behavior accumulates.

References

Google Machine Learning Course: Recommendation Systems Overview — Definition of recommendation models and two common forms
Google Machine Learning Course: Retrieval — Multiple retrieval sources and unified scoring
Google Machine Learning Course: Scoring — A single scoring model and position-bias correction
Google Machine Learning Course: Re-ranking — Constraints, freshness, and fairness during re-ranking
Item-to-Item Collaborative Filtering, IEEE Internet Computing, 2003 — Tens of millions of customers, millions of products, and the 0.5-second response constraint
ACM RecSys 2024 — The annual academic conference for recommendation systems

About Tengence

Recommendation systems are a core technology for converting public traffic and improving private-domain conversion efficiency, making them part of the growth technology landscape Tengence focuses on. Tengence is the enterprise growth engine for the AI era, focused on public-traffic acquisition and private-domain conversion.

Related Reading

Get Started