- A RAG answer has two failure modes that need separate meters: the retriever fetched the wrong context, or the generator mishandled the right context. One aggregate quality score hides which one you have.
- Measure retrieval with labeled queries (recall and precision at k, hit rate, MRR, context relevance) and measure generation with faithfulness, answer relevance, and correctness. The RAG triad ties the two together in three numbers.
- When an answer is wrong, localize it before you fix it: a retrieval miss points at chunking, embeddings, or reranking; a grounded-context error points at the prompt or the model. Build the eval set on your own data and wire it into CI.
Two ways a RAG answer fails
A retrieval-augmented generation system does two jobs in sequence. First it retrieves: given a question, it fetches a handful of passages from your corpus. Then it generates: given those passages and the question, it writes an answer. Because the jobs are sequential and distinct, so are the ways they break. The retriever can hand the generator the wrong context, or hand it nothing useful at all. Or the retriever can do its job perfectly and the generator can still ignore the context, contradict it, or answer a different question than the one asked. These are independent failures with independent causes and independent fixes.
The mistake nearly every team makes early is to measure only the thing the user sees: was the final answer good. That single verdict is real, but it is a lagging, blended signal. When it drops, it tells you something is wrong and nothing about where. You will spend a sprint rewriting the prompt to fix answers that were doomed the moment the retriever returned the wrong three passages, or swapping embedding models to fix a generator that was hallucinating over perfectly good context. The aggregate score cannot separate the two, so it cannot direct the fix.
The discipline that unlocks RAG evaluation is almost embarrassingly simple to state: instrument the two halves separately. Score whether retrieval fetched the right context, independently of whether the answer was good. Score whether the answer was faithful to and relevant given the context it received, independently of whether that context was the right context to fetch. Once those two meters exist, every regression sorts itself into a retrieval problem or a generation problem before you touch any code.
Measuring retrieval quality
Retrieval quality answers one question: did the system fetch the context that actually contains the answer. This half is the more tractable of the two, because retrieval is fundamentally a ranking problem and ranking has decades of well-understood metrics. The catch is that computing them honestly requires labeled data, that is, a set of representative questions each annotated with the passages in your corpus that answer them. There is no way around the labels; a metric that does not know the right answer cannot tell you whether retrieval found it.
With a labeled set in hand, the standard metrics fall into two families. Set metrics ask how much of the relevant material landed in the top k results, ignoring order: recall@k and precision@k, plus the coarser hit rate (did at least one relevant passage appear at all). Rank metrics reward putting the right passage near the top, since the generator weighs early context more heavily and context windows are finite: MRR (mean reciprocal rank) captures how high the first relevant hit sits. Alongside these, context relevance judges the fetched passages directly rather than against a label, and is often scored by an LLM at query time when labels are scarce.
| Metric | What it measures | What it tells you |
|---|---|---|
| Recall@k | Fraction of relevant passages found in top k | Is the answer even reachable downstream |
| Precision@k | Fraction of top-k passages that are relevant | How much noise the generator must wade through |
| Hit rate | Share of queries with at least one hit in top k | Coarse floor: how often retrieval fails outright |
| MRR | Reciprocal rank of the first relevant hit | Whether the best passage ranks high enough to matter |
| Context relevance | Judged relevance of fetched passages (no label) | Retrieval signal when labeled sets are thin |
For most enterprise RAG, recall@k is the metric to obsess over first, because it sets the ceiling: a passage the retriever never surfaces is a passage the generator can never use, and no prompt engineering recovers it. Precision and MRR then govern how efficiently that recalled context is presented. The upstream levers behind all three, splitting and embedding choice, are covered in Chunking and Embeddings.
Measuring answer quality
Retrieval quality stops at the boundary of the context window. Answer quality picks up where it ends and asks a different question: given the context that was retrieved, is the generated answer good. Holding the context fixed is the whole point. It isolates the generator so that a bad answer here is unambiguously a generation problem, not a retrieval one that leaked downstream. Three properties carry most of the weight, and they are worth keeping distinct because a generator can pass one while failing another.
Faithfulness, also called groundedness, asks whether every claim in the answer is supported by the retrieved context. This is the direct measure of hallucination in a RAG setting: an answer that asserts something the context does not say is unfaithful even if the assertion happens to be true in the world, because the system invented it rather than retrieved it. In regulated and high-trust domains, faithfulness is usually the metric with the sharpest business consequences, since an unsupported claim delivered with confidence is the failure mode that erodes user trust fastest.
Answer relevance asks whether the response actually addresses the question that was asked, rather than drifting into adjacent material the context happened to contain. A faithful answer can still be irrelevant: perfectly grounded, perfectly sourced, and about the wrong thing. Correctness is the third and most demanding property, comparing the answer against a known ground-truth answer where one exists. Correctness needs reference answers and so is more expensive to maintain, but for questions with a definite right answer it is the closest proxy to what the user cares about.
Faithfulness and answer relevance can both be judged from the question, the context, and the answer alone, with no reference answer required, which is exactly why they scale to large evaluation runs. That property is what makes automated judging practical, and it is the bridge to the framework in the next two sections.
The RAG triad
The metrics above are numerous enough to feel like a checklist, and checklists get skipped. The RAG triad compresses them into three relationships that together cover the whole pipeline, and its value is precisely that it maps to the three edges of the RAG data flow. Question, context, and answer are the three nodes; the triad scores the three connections between them.
QUESTION
/ \
/ \ answer relevance
context \ (answer vs question)
relevance \
(context vs \
question) \
\ \
CONTEXT ----- ANSWER
groundedness
(answer vs context)
Context relevance (context against question) is the retrieval-side edge: did the fetched passages actually bear on what was asked. Groundedness (answer against context) is the faithfulness edge: is every claim traceable to the passages provided. Answer relevance (answer against question) is the output edge: does the response address the original question. A failure on each edge localizes cleanly. Weak context relevance is a retrieval problem, full stop. Weak groundedness with strong context is a generation problem, the model departing from good source material. Weak answer relevance with strong context and groundedness usually means the answer is faithful to context that was retrieved but tangential, which points back at retrieval again.
The triad does not replace the detailed metrics; it organizes them and gives teams a shared vocabulary that survives contact with a status meeting. When someone reports "the RAG got worse," the useful follow-up is now mechanical: which edge of the triad moved. That question has an answer, and the answer routes the work.
LLM-as-judge for RAG
Two of the three triad edges, groundedness and answer relevance, can be assessed from the question, context, and answer with no reference label. That is what makes them scalable, and the scaling mechanism in practice is an LLM acting as judge. You prompt a capable model with the question, the retrieved context, and the generated answer, and ask it to rate whether each claim is supported and whether the response is on point, ideally returning a structured verdict with a short rationale. Run across an evaluation set, this turns two of the most important RAG qualities into numbers you can trend nightly, at a cost far below human annotation.
The caveats are real and you should design around them rather than pretend they are absent. LLM judges carry biases, toward longer answers, toward answers that echo their own phrasing, and toward surface fluency over substance. They are stochastic, so the same answer can score differently across runs unless you pin decoding low. And a judge is itself a model that can be wrong, so an unaudited judge quietly becomes an unmeasured single point of failure. The mitigations are the same discipline you would apply to any evaluator: calibrate the judge against a few hundred human-labeled examples, keep the rubric narrow and concrete, prefer clear binary or low-cardinality scales over vague 1-to-10 ratings, and periodically re-audit as models change underneath you.
This is why the division of labor matters. Use labeled query sets for the retrieval half, where ground truth is a matter of which passages contain the answer and human labels are both feasible and durable. Use LLM judges for the generation half, where the qualities are reference-free and volume makes human scoring impractical. The general practice of building, calibrating, and gating on judges, along with when to trust them, is treated in depth in the Evals deep-dive; RAG simply applies it to the triad's two reference-free edges.
Decomposing a bad answer
All of the machinery so far exists to serve one operational moment: an answer came back wrong, and you need to know why before you spend a day fixing the wrong thing. The decomposition is a short decision procedure run against the two meters you have built. Take the failing query, look at what was retrieved, and ask the retrieval question first, because it gates everything downstream: was the passage that answers this question present in the retrieved context at all.
If the answer is no, you have a retrieval miss, and no amount of prompt work will help, because the generator never had the material. The fix lives upstream, and which upstream lever depends on the shape of the miss. If the answering passage exists in the corpus but never surfaced, suspect embeddings or the query, or add a reranking stage. If it surfaced but ranked too low to fit the context budget, tune reranking or k. If the answer was split across a boundary so no single chunk carried it, the problem is chunking. If the answer is yes, the passage was right there in the context, then you have a generation error, and the fix is the prompt, the model, or the output format, never the retriever.
Doing this at scale is why the separate meters pay off. Rather than triaging failures one at a time, you can partition an entire evaluation run: queries with low context relevance go in the retrieval bucket, queries with good context but low groundedness or answer relevance go in the generation bucket. The relative size of the two buckets tells you where the next sprint should go, and the composition within each bucket tells you which specific lever to pull. Anecdotes become a ranked backlog.
The architect view
The recurring lesson across RAG programs is that the eval set is the asset, and almost everyone builds it too late. Stand up a modest evaluation set on your own data early, before the pipeline is tuned, ideally before it is even good. A hundred to a few hundred representative questions, each labeled with the passages that answer them and, where feasible, a reference answer, is enough to start. It does not need to be large to be decisive; it needs to be yours. Public benchmarks and vendor demos measure someone else's corpus and someone else's questions, and they will mislead you about your own.
From that foundation, three practices compound. Measure retrieval and answer quality on separate meters so every regression is pre-sorted into the half that broke. Wire the eval set into CI so that a chunking tweak, an embedding swap, or a prompt edit is gated by recall and groundedness the way code is gated by tests, rather than blessed by a demo that asked three friendly questions. And treat the eval set as living: fold in real failures from production logs, so the set grows toward the questions users actually ask.
Frameworks exist to compute these metrics, and RAGAS-style tooling can save you the plumbing of wiring judges and metrics together. Use them for the scaffolding, but do not confuse the framework with the evaluation. The framework computes numbers; the labeled questions and the reference answers, drawn from your corpus and your users, are what make those numbers mean anything. A team that owns its eval set, separates the two halves, and gates on both will improve its RAG system with evidence while competitors argue from anecdote. You can watch retrieval behavior shift as parameters change in the RAG Explorer, and the same separation of concerns underlies the wider evaluation practice in the Evals deep-dive.