RAG Evaluation: Metrics and Test Design

TL;DR: RAG evaluation checks two things: whether a retrieval augmented generation system pulls the right context, and whether its answers stay grounded in that context. Score retrieval and generation separately against a fixed, versioned question set. For retrieval, use context precision and recall. For generation, use faithfulness, answer relevance and citation accuracy. Retrieval on its own does not stop hallucination. One study found that retrieval based legal research tools still hallucinated on 17% to 33% of queries (Magesh et al., Stanford, 2024). Rerun the same suite on every model, prompt or index change. ContextQA, the autonomous quality engineer, tests agent responses and guardrails without an SDK or code access.

A lot of RAG testing stops at "the answer looked right." That check misses retrieval failures, ungrounded claims and regressions that appear after a model upgrade. If your retrieval augmented generation (RAG) pipeline ships through CI/CD, you need checks that can fail. This guide covers retrieval and grounding metrics, fixtures with known answers, seeded failures and release gates.

Quick answers

How do you add RAG checks to CI/CD without flaky releases? Use two tiers. Deterministic checks gate every merge: known document IDs for fixed queries, required citations and refusals on out of scope questions. Model graded scores such as faithfulness run nightly or before release against a versioned dataset. Set thresholds from a recorded baseline. The baseline does not remove run to run variance, but it makes a genuine drop easier to tell apart from noise.

What should trigger a full RAG regression run? Rerun the full suite after any change to the embedding model, chunking strategy, top k, reranker, prompt template, LLM version or a large re ingestion. Any of these can change answers with zero code changes. Log the dataset version, model version and index snapshot for every run so reviewers can trace each score change to its cause.

How do you test RAG guardrails for PII leakage and cross tenant data? Seed the corpus with synthetic records that carry known canary values, such as fake account numbers or documents tagged by tenant. Send adversarial prompts that try to extract them, including from another tenant's session. Assert that no canary string appears in the retrieved context or the answer. Keep this as a separate blocking suite, because one leak should stop a release.

What is RAG evaluation and when do you need it?

This kind of evaluation measures whether a retrieval augmented generation system retrieves the right documents and turns them into accurate, grounded answers. It scores the retriever and the generator separately, so you can trace a bad answer to one stage. You need it before launch and after every prompt, chunking, embedding or model change.

Most teams track four metrics:

• Faithfulness: Every claim in the answer is supported by the retrieved context. A low score means the model is hallucinating, stating things the sources do not say.
• Answer relevancy: The answer addresses the question the user asked, not a nearby one.
• Context precision: Relevant chunks rank near the top instead of sitting under noise.
• Context recall: The retrieved set contains every fact the reference answer needs. When recall is low, the generator never had the facts.

A reindex can lower recall while answers still read well. Write the reference answers yourself, not with the system under test.

What do you need before you start?

You need four things: a versioned test set of questions, each with known relevant document IDs and a reference answer; access to the retriever's ranked results; a chosen set of retrieval and answer metrics; and a frozen baseline run. Without gold document IDs, you cannot compute ID based metrics such as hit rate, recall@k, MRR or nDCG. Reference free metrics still work without them, but gold IDs give you deterministic checks that can gate a merge.

Three kinds of tooling fit here:

• Open source metric libraries: These score faithfulness and context precision in Python. You own the harness, the judge model and the CI wiring.
• Tracing platforms: These record each retrieval and generation span for debugging. You still build the labeled dataset.
• Black box agent testing: ContextQA AI agent testing checks responses, guardrails and tool calls without an SDK or code access, and reruns scenarios after model upgrades.

Score retrieval against the gold IDs with these metrics:

• Hit rate: The share of queries with at least one relevant chunk in the top k.
• Recall@k: The fraction of all relevant chunks that appear in the top k.
• MRR (mean reciprocal rank): The average of 1 divided by the rank of the first relevant chunk.
• nDCG (normalized discounted cumulative gain): Gives more credit to relevant chunks that rank higher, and supports graded relevance.

Build the labeled dataset first. Every tool's scores depend on it.

How do you run a RAG test suite step by step?

Start with a golden dataset: questions with known answers and known source documents. Run each question through the pipeline, then score retrieval and generation separately. A judge model, an LLM prompted with a scoring rubric, rates faithfulness, and any claim the retrieved context does not support counts against the case's threshold.

1. Build the golden set. Pull real questions from production logs and pair each with its expected answer and supporting docId values. Add synthetic cases generated from your docs and policies. A domain owner reviews every case.
2. Score retrieval. Assert that the expected documents appear in the top results. This check is deterministic, so it can gate every merge.
3. Judge faithfulness. Send the question, context and answer to the judge, which marks each claim supported or unsupported. Before you trust its scores, compare a sample of its verdicts with human labels.
4. Gate in CI. Run the judge scored suite nightly or before release, because LLM judged scores vary between runs. Fail the gate when unsupported claims exceed the baseline threshold, and rerun after every model upgrade.

import { test, expect } from '@playwright/test';

import golden from './golden.json';

for (const c of golden) {

test(`rag ${c.id}`, async ({ request }) => {

const body = await (await request.post('/api/ask', { data: { q: c.question } })).json();

expect(body.sources.map((s: any) => s.docId)).toEqual(expect.arrayContaining(c.expectedDocIds));

// Judge scored check: run in the nightly or pre release project, not on every merge.

const verdict = await (await request.post('/api/judge', {

data: { question: c.question, context: body.context, answer: body.answer },

})).json();

expect(verdict.unsupportedClaims.length).toBeLessThanOrEqual(c.baselineMaxUnsupported);

});

}

What are the most common mistakes?

Four mistakes come up again and again: scoring only the final answer, trusting an LLM judge without human calibration, freezing the question set, and running checks by hand before releases. Each one lets retrieval or grounding regressions reach production, because nothing fails when quality drops.

• Answer only scoring: A fluent answer can hide a retrieval miss. Check that the expected chunk was retrieved and that each claim traces back to it.
• Uncalibrated judges: LLM judges drift when prompts or models change. Compare a fixed human labeled sample with the judge's verdicts, and recalibrate when they disagree.
• Frozen golden set: Add a case for every production failure and every new document type.
• Manual pre release runs: Run deterministic checks in CI on every change, run model graded scores nightly or before release, and fail the build when a threshold is breached.

How do you know it is working?

Your evaluation works when a fixed, versioned question set catches regressions before users do. Retrieval metrics flag missing context, and faithfulness scores flag unsupported answers. If a seeded bad document or a deliberately broken retriever does not fail the gate, the evaluation is not working.

In production, monitor three signals separately:

• Drift: Score a daily sample of live traces for faithfulness and context_recall. Alert when scores stay below the release baseline.
• Latency: Track p95 (the 95th percentile response time) for retrieval and generation separately, so a slow vector index does not get blamed on the model.
• Cost: Log tokens per answer. Extra retrieved chunks raise cost, and whether they improve accuracy depends on your pipeline, so measure both together.

Keep a ledger of the model upgrades and reindexes the gate blocked. That ledger shows what the work is worth.

How can you automate RAG evaluation with ContextQA?

ContextQA tests your retrieval augmented agent from the outside. AI agent testing generates scenarios from the documents and policies your agent relies on, then checks responses, guardrails and tool calls. It needs no SDK, instrumentation or code access, and it also covers red teaming, PII and compliance checks, and model upgrade regression.

A practical workflow:

1. Generate scenarios from the source corpus. Point scenario generation at the docs and policies your retriever indexes.
2. Run against the deployed agent. This includes agents built on Agentforce, Bedrock and Cortex.
3. Assert on what users see. Check that answers stay within policy, that the expected tools get called and that no PII leaks.
4. Rerun after every model change. Model upgrade regression shows which answers changed after a provider or version swap.

Where metric scores fit: The product page does not list numeric RAG metrics such as context precision. Compute those in your own harness and confirm details during a demo.

Pilot framework: Set a baseline and stop conditions before the pilot starts, using the AI testing pilot scorecard.

If your RAG agent gives confident answers built on the wrong retrieved context, see how ContextQA tests agent responses, guardrails and tool calls without SDK or code access, or book a demo.

What should you do right now? (A checklist)

Freeze a labeled question set, score retrieval and generation separately, and gate releases on those scores. Record the baseline before you change a prompt, index or model, then rerun the same set on every change. A regression then shows up as a failed check, not a customer complaint.

• Create golden_set.jsonl with questions, expected source document IDs and reference answers written by a domain owner.
• Assert that expected document IDs appear in the top k results.
• Score faithfulness so every claim traces to a retrieved chunk.
• Add questions the corpus can't answer, and assert that the system declines instead of guessing.
• Plant one known bad chunk and confirm your metric flags it.
• For reference free metrics, read Es et al., 2023, arXiv.
• Run PII and policy checks against the deployed agent.
• Record each threshold and its owner in a decision log.

Bottom line

Test retrieval and generation as separate layers. Score retrieval on whether the right passages came back, and score answers on faithfulness and relevance. Lock both into a versioned fixture set, so a model upgrade or index change fails in CI, not in production. Give every manual metric a named owner. For the full workflow, see the RAG testing guide and agentic testing vs AI agent testing.

Sources

These are the primary sources for the hallucination statistic and the reference free metric definitions in this guide. Both are research papers on arXiv, linked so reviewers can check the method, datasets and numbers directly.

• Magesh et al., Stanford, 2024
• Es et al., 2023, arXiv