What is the RAG Triad Evaluation Framework?
The RAG Triad Evaluation Framework is a methodology for assessing retrieval-augmented generation systems across three measurable dimensions: Context Relevance, Groundedness (Faithfulness), and Answer Relevance. Unlike traditional software, RAG systems are non-deterministic, combining the variability of natural language queries, the stochastic nature of semantic search, and the unpredictable generative outputs of large language models. This complexity makes manual evaluation unscalable and traditional metrics like BLEU or ROUGE inadequate.
The term "RAG Triad" was coined by TruLens, the open-source evaluation library from TruEra (acquired by Snowflake in May 2024), and the same three-way decomposition appears in Ragas under slightly different metric names. The triad breaks down the RAG pipeline into three distinct, measurable dimensions. Rather than evaluating the final response as a monolith, it isolates the two core components of RAG, the retrieval system and the generative system, and maps their interactions to three relationships.
By isolating these variables, AI engineering teams can move beyond vague feedback like "the answer was wrong" to pinpointing where and why the system failed. Did the vector database fail to find the right information? Did the LLM ignore the provided context? Or did the LLM fail to directly address the user's original question? The RAG Triad provides a structured vocabulary and measurement methodology for answering these questions at scale, as part of a broader LLM evaluation practice.
The Three Pillars: Context Relevance, Groundedness, and Answer Relevance
The RAG Triad consists of three fundamental metrics, each evaluating a specific step and relationship within the RAG pipeline.
1. Context Relevance (User Query ↔ Retrieved Context)
What it measures: Context Relevance assesses whether the information retrieved by your search system (vector database, BM25, or hybrid search) is actually useful for answering the user's query.
Why it matters: This evaluates your retrieval component. If the retrieved context is irrelevant, the generative model has no factual basis upon which to construct a correct answer. It is the classic "garbage in, garbage out" problem. High context relevance means your chunking strategy, embedding model, and search algorithms are performing well.
How it works: An evaluator (often an LLM acting as a judge) analyzes the user's query and the retrieved document chunks. It scores each chunk on whether it contains information necessary to address the query. Often, this includes penalizing systems that retrieve too much irrelevant information, as excessive noise can confuse the generation model and bloat token costs.
2. Groundedness / Faithfulness (Retrieved Context ↔ LLM Response)
What it measures: Groundedness (sometimes called Faithfulness) determines if the final answer generated by the LLM is entirely derived from and supported by the retrieved context.
Why it matters: This is your primary defense against hallucination. An ungrounded answer might be factually correct in the real world, but if it wasn't derived from your proprietary retrieved context, the RAG system has failed its primary objective: restricting the LLM to verified knowledge. If an answer introduces new facts, numbers, or claims not present in the source documents, it lacks groundedness. See hallucination evaluation for methods specific to detecting this failure mode.
How it works: The evaluator compares the LLM's final response against the retrieved chunks. It checks every claim made in the response to ensure a direct lineage back to the source text. Responses that extrapolate, hallucinate, or bring in outside knowledge receive low groundedness scores.
3. Answer Relevance (User Query ↔ LLM Response)
What it measures: Answer Relevance evaluates whether the final generated response directly and fully addresses the user's original query.
Why it matters: Even if you retrieve perfect context and generate a completely grounded response, the system fails if it doesn't answer the user's actual question. This metric catches instances where the system goes off on a tangent, provides an incomplete answer, or gives a technically accurate but practically useless response.
How it works: The evaluator compares the initial query with the final response, disregarding the retrieved context for a moment. It asks: "Did this response satisfy the user's intent?" This helps identify issues with the final generation prompt or instances where the LLM became distracted by secondary information in the context.
Diagnosing Pipeline Failure Modes
The RAG Triad's value lies in its diagnostic capability. By looking at the combination of scores across the three pillars, engineering teams can identify failure modes and know which component of their pipeline needs optimization.
Here is a diagnostic decision table to help interpret RAG Triad results:
| Context Relevance | Groundedness | Answer Relevance | Diagnosis & Root Cause | Recommended Fixes |
|---|
| High | High | High | Success: The system is working perfectly. | Monitor to ensure continued performance. |
| Low | High | Low | Retrieval Failure: The system retrieved the wrong documents. The LLM faithfully summarized those wrong documents, resulting in an irrelevant answer. | Improve chunking, try a different embedding model, implement hybrid search, or add query expansion. |
| High | Low | High | Hallucination / Knowledge Leakage: The system found the right docs, but the LLM ignored them and used its internal weights to answer the question. | Strengthen the system prompt (e.g., "Answer ONLY using the provided context"), use a better/larger model, or lower temperature. |
| High | High | Low | Prompt / Generation Failure: The right docs were found, and the LLM stuck to them, but it failed to actually synthesize an answer to the specific question asked. | Refine the generation prompt to better guide the synthesis. Ensure the retrieved context isn't too long or confusing. |
| Low | Low | High | Lucky Guess (Dangerous): Bad retrieval, but the LLM knew the answer anyway. Highly prone to future hallucinations. | Treat as a retrieval failure. Do not trust the system just because the answer happens to be right this time. |
| Low | Low | Low | Catastrophic Failure: The pipeline is broken at every stage. | Re-evaluate the entire architecture. Start by fixing retrieval first. |
How Ragas and TruLens Compute These Scores
The triad's three relationships map onto named metrics in the two frameworks that popularized it, and knowing the actual computation helps you interpret the numbers.
| Triad pillar | TruLens name | Ragas name (current docs) | How Ragas computes it |
|---|
| Context Relevance | Context Relevance | Context Precision | LLM judges each retrieved chunk as useful or not, averaged as precision@k over the ranked list, so relevant chunks ranked higher score better |
| Groundedness | Groundedness | Faithfulness | LLM decomposes the answer into individual claims, then checks each claim against the retrieved context; score = supported claims / total claims |
| Answer Relevance | Answer Relevance | Response Relevancy | LLM generates several questions the answer would be answering, embeds them, and averages their cosine similarity to the actual query |
Two details are easy to miss. First, all three triad metrics are reference-free: they need no ground-truth answers, which is what makes them usable on live production traffic. Ragas also offers reference-based metrics, most importantly context recall (did the retrieved context contain everything the reference answer needs?), which requires a labeled dataset and therefore lives in offline evaluation. Second, Ragas added noise sensitivity, which measures how often irrelevant retrieved chunks induce wrong claims in the answer; it catches a failure the triad misses, where retrieval is technically relevant but the extra noise still derails generation.
The faithfulness computation also explains a common surprise: a one-sentence answer decomposes into one or two claims, so a single unsupported claim can swing the score from 1.0 to 0.5. Aggregate scores over hundreds of examples before drawing conclusions.
Automated Scoring Prompts & Evaluation Rubrics
To scale the RAG Triad, teams typically use "LLM-as-a-judge" techniques. This involves using a highly capable model (such as GPT-5, Claude Opus 4.5, or Gemini 3 Pro) to evaluate the outputs of the primary RAG pipeline based on strict grading rubrics.
Below are examples of how you might structure the evaluation prompts and rubrics for each pillar.
Context Relevance Rubric
Input to Evaluator: User Query, Retrieved Document Chunk
Prompt Structure:
You are an expert relevance assessor. Your task is to evaluate if the provided Document Chunk contains information that is necessary to answer the User Query.
Assign a score from 1 to 5 based on this rubric:
1: The chunk is completely irrelevant to the query.
2: The chunk shares some keywords but provides no useful information to answer the query.
3: The chunk provides tangential information that might be slightly helpful but is not central to the answer.
4: The chunk contains useful information that directly helps answer a part of the query.
5: The chunk contains the exact, comprehensive information needed to fully answer the query.
Output your evaluation in JSON format containing "score" (integer) and "reasoning" (brief explanation).
Groundedness (Faithfulness) Rubric
Input to Evaluator: Retrieved Context (All Chunks), LLM Generated Response
Prompt Structure:
You are a strict fact-checker. Your task is to verify that every claim made in the Generated Response is directly supported by the Retrieved Context. The response must not introduce any outside information.
Assign a score from 1 to 5 based on this rubric:
1: The response is completely ungrounded. It relies entirely on outside knowledge or hallucination.
2: Major claims in the response are unsupported by the context, though some minor points are grounded.
3: The response is mostly grounded, but includes a few minor unsupported claims, numbers, or extrapolations.
4: The response is fully grounded in the context, but may mildly overstate or slightly misinterpret a minor detail.
5: The response is perfectly faithful. Every single claim, fact, and figure is directly traceable to the provided context.
Output your evaluation in JSON format containing "score" (integer), "unsupported_claims" (list of strings), and "reasoning" (brief explanation).
Answer Relevance Rubric
Input to Evaluator: User Query, LLM Generated Response
Prompt Structure:
You are an intent fulfillment evaluator. Your task is to determine if the Generated Response directly and comprehensively answers the User Query. Do not evaluate factual accuracy, only whether the response addresses the prompt.
Assign a score from 1 to 5 based on this rubric:
1: The response completely ignores the query or answers a totally different question.
2: The response touches on the topic but fails to address the core intent of the query.
3: The response partially answers the query but misses significant components or is unnecessarily evasive.
4: The response directly answers the core query but includes excessive fluff or misses a minor sub-question.
5: The response is concise, direct, and comprehensively answers all parts of the user's query.
Output your evaluation in JSON format containing "score" (integer) and "reasoning" (brief explanation).
Continuous Evaluation in CI/CD vs. Online Production Telemetry
Implementing the RAG Triad effectively requires applying it at two different stages of the application lifecycle: during development/deployment (offline) and during live production (online).
Offline Evaluation (CI/CD Pipelines)
Before deploying a new embedding model, tweaking a prompt, or changing your chunk size, you need to prove the change improves the system. Offline evaluation involves running the RAG Triad over a static, golden dataset of hundreds or thousands of query-context-response pairs.
- Execution: Run asynchronously as part of GitHub Actions or CI pipelines.
- Scale: Exhaustive evaluation across a large dataset to calculate aggregate scores (e.g., average groundedness of 4.2).
- Models: You can afford to use the most expensive, capable LLMs (like GPT-5 or Claude Opus 4.5) as judges because this is done periodically, not on every user request.
- Goal: Prevent regressions. Ensure that a change intended to fix one edge case doesn't break performance across the rest of the dataset.
Online Evaluation (Production Telemetry)
Once in production, you need to know how the system is performing on real, unforeseen user queries. Online evaluation involves calculating RAG Triad scores on live traffic.
- Execution: Run asynchronously in the background via message queues (e.g., Kafka, Celery) so it doesn't impact user latency.
- Scale: Often applied to a sampled subset of traffic (e.g., 5% of queries) due to cost and volume.
- Models: Typically uses smaller, faster, and cheaper LLMs (like GPT-5 mini, Claude Haiku 4.5, or specialized fine-tuned judges) to keep costs manageable at high volume.
- Goal: Detect drift, identify new failure modes, and flag poor responses for human review. Low-scoring interactions in production should be funneled back into the offline golden dataset for future testing.
Frequently Asked Questions
Is "LLM-as-a-Judge" reliable enough for the RAG Triad?
While not infallible, research has shown that highly capable LLMs exhibit strong correlation with human annotators when provided with clear, structured rubrics. To improve reliability, teams often use techniques like self-consistency (asking the judge multiple times and taking the majority vote) or fine-tuning smaller models specifically on human-graded datasets to act as specialized evaluators.
Do I have to calculate all three metrics for every query?
In development (offline evaluation), yes, you should calculate all three to fully understand system performance. In production (online evaluation), you might choose to prioritize based on your specific risks. For example, in a medical or legal application, you might sample Groundedness more heavily to aggressively monitor for hallucinations, while in a customer support bot, Answer Relevance might be the primary focus.
How do I handle multi-turn conversations with the RAG Triad?
Evaluating multi-turn chat adds complexity because the context and intent evolve. In these scenarios, the "User Query" passed to the evaluator should often be a synthesized or rephrased version of the entire conversation history leading up to the final turn, rather than just the user's last immediate input. This ensures the Answer Relevance metric accurately judges whether the system responded appropriately within the context of the ongoing dialogue.