Back

Building RAG Evaluation Metrics: A Two-Layer Quality Gate From Retrieval nDCG to Generation faithfulness

Tengence Team ·
RAG Search System Tengence Search
Building RAG Evaluation Metrics: A Two-Layer Quality Gate From Retrieval nDCG to Generation faithfulness

Summary: After shipping a retrieval-augmented generation system, many teams are left with only a subjective impression of "answer accuracy": tuning by instinct, regression checks by sampling. A more reliable approach splits evaluation into two layers. Retrieval asks whether the right documents were found and placed in the right positions; generation asks whether the answer stays within those documents. Drawing on the official Ragas metric docs, the Microsoft Learn retrieval-quality methodology and the Elasticsearch ranking evaluation API, this article sets out the definitions, boundaries and common misreadings of each layer, then shows how to close the loop with judgment lists, A/B reports and operational intervention.

Key Takeaways

1. Split Evaluation Into Two Layers Instead of Chasing One Composite Score

The most common mistake in RAG systems is collapsing retrieval and generation into a single composite score. The cost is loss of diagnosability: when the score drops, no component can be held responsible. The official Ragas evaluation guide separates metrics explicitly. The retriever side offers context_precision and context_recall to measure the retrieval system; the generator side offers faithfulness, which measures hallucination, and answer_relevancy, which measures whether the answer addresses the question. This is not a naming convention but a precondition for fault localization: low context precision means retrieval or reranking pushed noise into the prompt; low faithfulness means the model supplied facts beyond the context.

The structure of the evaluation data follows from that split. A Ragas test set is a triplet of question, context and ground_truth: the real query, the retrieved passages actually sent to the model, and a human-written reference answer. With this in place, evaluate output can be converted into a table for case-by-case inspection, revealing which question types are systematically weak. The Red Hat evaluation-systems documentation calls this evaluation-driven development: once the test set is frozen, chunking strategy, embedding model and retrieval algorithm become comparable variables instead of a configuration chosen by intuition.

For systems centred on on-site search, the split carries an additional business meaning. When users search products, documents or help-centre entries, retrieval metrics map directly to "can it be found"; faithfulness only starts to matter once generative answers are attached, where it maps to "will it invent something". An enterprise-grade on-site search engine makes query understanding, multi-path retrieval and ranking independently tunable precisely so that these two layers can be measured and intervened on separately. Compressing them into one score abandons that controllability.

2. Retrieval: Why DCG@10 Is Recommended Over Hit Rate Alone

The retrieval layer does not lack metrics; it lacks selection criteria. The Microsoft Learn retrieval-quality guide supplies a complete set of definitions. Recall@k is the fraction of known relevant documents appearing in the top-k results, suited to RAG scenarios where missing one item leaves the answer incomplete. Precision@k is the fraction of the top-k results that are relevant, suited to search interfaces where irrelevant hits erode trust. MRR is the average of 1/rank across queries, the reciprocal of the position of the first relevant result, suited to question answering where the first result decides the experience. MAP@k computes precision at each relevant result position and averages, capturing overall ranking quality.

DCG and NDCG differ in whether they are sensitive to both position and graded relevance. The guide uses a 0 to 3 scale with gain computed as 2^rel − 1, so a score-3 result contributes 7 while a score-1 result contributes 1, and lower positions are discounted more heavily. Two benchmarks from the guide are especially useful for interpretation: the theoretical maximum DCG@10 when all results score 3 is 31.80; if all 10 results score 2, DCG@10 is only 13.63, and a 1-point gain is roughly a +7% relative improvement. Score changes can therefore be translated into business language — approximately one result on the page becoming noticeably better.

The NDCG trap deserves separate emphasis. Dividing DCG by the ideal DCG normalizes the value to 0 to 1, which makes queries comparable but erases total information volume. The Traditional Chinese edition of the Microsoft Learn guide offers a contrast: result sets A and B both have NDCG@5 of 1.00 because each is sorted by descending relevance, yet their DCG@5 values are 4.26 and 8.02, with the latter delivering nearly twice the useful information. In other words, "two good results among three irrelevant ones" and "five good results" look identical under NDCG. That is why the guide recommends DCG@10 rather than NDCG as the primary metric: graded relevance preserves nuance, positional discounting matches browsing behaviour, and absolute utility reflects how much useful information users actually received.

Metrics also need judgment lists to be computable. The Elastic search-relevance documentation explains that a judgment list is only an expectation about query results; its value comes from feeding it, together with the query and a chosen metric, into a ranking evaluation API to obtain an objective score for precision, recall, DCG and MRR. The Elasticsearch _rank_eval documentation goes further on request structure: mean_reciprocal_rank accepts k (for example 20) and relevant_rating_threshold, dcg accepts k and a normalize switch, and ERR is based on a cascade-model assumption, fitting question answering and navigation queries but less suited to survey-style needs where users scan many results. The quality of the judgment list therefore sets the ceiling for the entire evaluation: with a skewed label distribution, even random ordering can produce a deceptively high score.

3. Generation: The Boundary Between Faithfulness and Answer Relevancy

The two core generation metrics answer different questions. The Ragas Faithfulness documentation is explicit about the calculation: break the response into independently verifiable claims, check each claim against the retrieved context, then divide the number of supported claims by the total number of claims, yielding a value from 0 to 1. Its example has only one of two claims supported by the context, giving 0.5. This turns "hallucination" from a subjective impression into a countable proportion and makes localization concrete — whether a question type routinely lacks key passages, or the model habitually appends background knowledge.

Answer relevancy measures whether the response addresses the question. Ragas derives several synthetic questions from the answer and averages their semantic similarity to the original question. Both depend on an LLM as judge, which is why the Red Hat documentation stresses engineering constraints: run evaluation on a frozen test set, set quality gates, retain evaluation history, and distinguish inline from remote provider modes — the former for small-sample development checks, the latter for distributed batch evaluation inside a pipeline integrated with CI/CD. The cost and bias of the judging model itself also need to be controlled through a fixed scoring policy and sampled human review.

One common misuse is treating low context_recall as a generator capability problem. It measures whether retrieval brought back all information needed to answer, so the responsibility lies with the index, recall or reranking. Likewise, low context_precision means noise entered the prompt, and even with good passages present the model may be pulled toward irrelevant facts. Reading these two alongside faithfulness and answer relevancy produces a responsibility matrix: whichever layer loses points is the layer to tune, rather than reaching for a larger model by default.

4. From Scores to a Loop: Judgment Lists, A/B Reports and Operational Intervention

An evaluation system only generates lasting value when it is wired into the release process. A workable sequence is: build judgment lists from the real query distribution, covering head terms, long-tail terms, spelling corrections and synonyms; freeze the test set and scoring policy and set a release threshold for each layer; rerun on the same test set and retain versioned scores whenever chunking, embeddings, recall channels or fine-ranking weights change; finally validate with online A/B reports whether offline gains actually convert into clicks and conversions. The Red Hat documentation groups this under evaluation-driven development and automated quality gates, aimed precisely at preventing silent regressions from configuration changes.

For on-site search, offline metrics and online behaviour also need an operational intervention layer between them. Query understanding — tokenization, stop words, spell correction, synonyms, category prediction and term weighting — determines whether long-tail and misspelled queries reach the right candidate set. Multi-path retrieval across inverted indexes, AI semantics and vectors determines whether a single channel's blind spot blocks the result. Ranking, with coarse and fine ranking models plus pinned positions, blocking, direct answers, result diversification and boosting, determines whether evaluation findings can quickly act on specific queries. Reporting and A/B experiments then place "DCG@10 improved for this query class" next to "click-through changed for this class" in one view. Without this layer, evaluation remains an offline report that never converges into reproducible tuning actions.

It is worth stating plainly that this metric system serves on-site retrieval and question-answering quality, measuring whether users can find and trust content within a site. Brand visibility in external search engines or generative answers is a different problem. Mixing the two into one metric set removes the reference needed to set thresholds: the former has judgment lists as ground truth, while the latter depends on how external platforms index and cite content. Keeping the evaluation target distinct is what makes the metrics interpretable.

FAQ

Q: What is wrong with evaluating a RAG system using one composite score?

A: You cannot localize the responsible component when the score drops. Retrieval and generation fail in different ways — the former is fixed by better recall and reranking, the latter by constraining generation and supplementing context — so a merged score removes the direction for tuning.

Q: Why does a high NDCG not mean the result set is actually useful?

A: Dividing by the ideal DCG normalizes the value to 0 to 1 and erases total information volume. Two result sets can both show NDCG@5 of 1.00 while DCG@5 is 4.26 and 8.02, so DCG@10 is recommended as the primary metric alongside absolute utility.

Q: Does low faithfulness always mean the model is hallucinating?

A: Not necessarily. If retrieved passages lack key facts, the model may append statements to complete the answer, which also lowers faithfulness. Check context_recall first to rule out a recall gap before deciding whether generation needs constraints.

Q: Should MRR or Recall@k take priority?

A: It depends on the interaction shape. Question answering where the first result decides the experience calls for MRR; scenarios where missing any relevant document leaves the answer incomplete call for Recall@k; search interfaces where irrelevant hits erode trust call for Precision@k.

Q: How large should a judgment list be, and how is it maintained?

A: Prioritize coverage of the real query distribution — head terms, long-tail terms, spelling corrections and synonyms — and refresh it as query logs and business categories change. With skewed label distributions, random ordering can score deceptively high, so inspect the distribution rather than the mean alone.

Data Sources

Related Reading

Get Started

About TENGENCE

TENGENCE is an AI-era enterprise growth engine focused on public-domain acquisition and private-domain conversion. Its core products are TENGENCE GEO (GEO + SEO dual engine) and TENGENCE Search. For teams that need retrieval quality to be measurable and intervenable, TENGENCE Search is an enterprise-grade on-site search engine: query understanding (tokenization, stop words, spell correction, synonyms, category prediction and term weighting), multi-path retrieval (inverted index, AI semantic and vector), ranking (coarse and fine ranking models, plus pinned positions, blocking, direct answers, result diversification and boosting), and reporting linked to A/B experiments. It competes directly with Algolia, Alibaba Cloud OpenSearch, OpenSearch, Elasticsearch and Unbxd, supports high-concurrency, low-latency retrieval, and also serves RAG and vector-retrieval knowledge-base QA systems. The two-layer metrics and judgment-list method described here are the engineering basis such systems use to measure "found" and "stated correctly" separately.

← Previous: Funnel Analysis Methodology: How to Precisely Locate Drop-off Points Next: What to Do When E-Commerce Conversion Rate Is Low: Diagnostic Checklist and Walkthrough →
Consult
QR code

WeCom