How to evaluate a RAG system without fooling yourself

Most RAG systems are tested the same way at first: someone asks ten questions, reads the answers, and says they look fine. That works until the first real complaint, when nobody can say whether last week's change made answers better.

On the on-prem RAG system I build at HealthCare Innovation, the question I wanted to answer on every release was simple: are the answers still grounded in the documents, and are they still the right documents? This post is how I measure that.

Start with a reference set you can version

Every evaluation needs questions with known good answers. Ours is a set of real questions from users, each with:

• the question as the user typed it, typos included;
• the passages that should be retrieved;
• a short reference answer written by someone who knows the domain.

The set lives in the repository next to the code and gets a version number. When the documents change, the set changes with them, in the same pull request. An evaluation score without the version of the set it was measured on means nothing.

Measure retrieval and generation separately

A RAG answer can go wrong in two places, and one score hides which one. I track them apart:

RAGAS gives a reasonable first version of each metric. For the cases it scores badly, mostly long procedural answers, an LLM judge with a written rubric does better. I keep both, and I read a sample of the judge's decisions myself every release, because a judge can drift too.

Turn the scores into a release gate

Scores on a dashboard get looked at once. Scores that can block a release get looked at every time. The gate is a few lines of code in the release pipeline:

THRESHOLDS = {"faithfulness": 0.90, "context_recall": 0.80}

def release_gate(scores: dict[str, float], baseline: dict[str, float]) -> list[str]:
    """Return the reasons to block this release, or an empty list."""
    failures = []
    for metric, floor in THRESHOLDS.items():
        if scores[metric] < floor:
            failures.append(f"{metric} {scores[metric]:.2f} is under {floor}")
        if scores[metric] < baseline[metric] - 0.02:
            failures.append(f"{metric} fell from {baseline[metric]:.2f}")
    return failures

Two rules, not one: a floor that no release may go under, and a limit on how much any metric may fall compared with the last release. The second rule catches slow decay that a fixed floor never would.

Trace the failures, don't average them

An average hides the questions that broke. Every evaluation run is traced in Langfuse, so a failed question opens to the exact passages retrieved, the prompt, and the answer. Most fixes I made came from reading ten failed traces, not from looking at a mean.

The useful output of an evaluation is not a number. It is a short list of questions that got worse, with the reason for each.

What changed

With this loop in place, answer relevance on the reference set went up by 25% across more than 10,000 documents, and every change since has shipped with a number attached. The tools matter less than the habit: a versioned set, two kinds of metrics, a gate that can say no, and traces you actually read.