Evaluating Retrieval and Generation: From Evidence to Correct Answers
Fluent answers do not prove correct retrieval, and an existing citation does not necessarily support its neighboring claim. Evaluate finding evidence separately from answering with it, then test their combination. The Ragas paper distinguishes retrieved context, answer faithfulness, and generation quality, and explores model-assisted scoring. The Python library of the same name is still on PyPI, but as of 2026-10-29 its latest release is still 0.4.3, published on 2026-01-13.
Define an evaluation case
For a fictional equipment-manual assistant, store the question, reference answer, applicable manual version, supporting passage locations, required conditions, and expected behavior when evidence is absent. Answers can have multiple valid phrasings; evidence and essential facts are more stable than one reference sentence.
Include exact identifiers, paraphrases, multi-passage questions, conflicts between old and current versions, and unanswerable questions. Draw from intended use. Questions generated by one model directly from documents may echo the source vocabulary and underestimate real users' phrasing. Generated additions still need sampled human checks of answerability and evidence.
Keep near-duplicates in the same split so that tuning questions do not return as paraphrased test cases. Freeze the corpus snapshot: a previously unanswerable question becomes a different test when a new document answers it.
Measure recall and ranking separately
If a question has 3 labeled relevant passages and 2 occur in the first 5 results, Recall@5 = 2/3. Relevant passages may duplicate one another or provide separate necessary facts. Also track whether the evidence contains all facts needed to solve the question, rather than rewarding repeated content.
For index quality, measure ANN recall against exact neighbors separately from this recall of relevant evidence.
MRR measures the position of the first relevant result. First hits at positions 1, 4, and no hit give (1 + 1/4 + 0) / 3 ≈ 0.417. It suits finding one useful result quickly but cannot establish completeness for multi-evidence questions. nDCG supports graded relevance and rewards valuable early results, while still depending on labels.
rankings = [["a", "b", "c"], ["x", "y", "z"], ["m", "n", "o"]]
relevant = [{"a", "c"}, {"z"}, {"p"}]
recalls = [len(set(r) & truth) / len(truth)
for r, truth in zip(rankings, relevant)]
print(round(sum(recalls) / len(recalls), 3)) # 0.667
This computes macro-average Recall@3 for three invented questions, weighting each equally. With incomplete relevance judgments, describe recall over labeled relevant items; do not treat every unlabeled passage as irrelevant.
Distinguish faithful, correct, and complete
Faithfulness asks whether supplied context supports the claims. Correctness asks whether claims match applicable facts. Completeness asks whether necessary conditions are included. An answer can faithfully repeat an obsolete manual and still be wrong for the current version. One correct but irrelevant sentence is not a complete answer.
Check both citation precision—does the cited passage support the claim?—and evidence coverage—are claims needing support actually sourced? A long list of links establishes neither. On unanswerable questions, separately count incorrect answers and appropriate abstention so that guessing does not inflate the apparent completion rate.
Check a model judge against hand labels first
Having another large model grade the answers (LLM-as-judge) works for screening at scale, but the judge has biases of its own. The MT-Bench and Chatbot Arena study names three: position bias (favouring the answer in a given position), verbosity bias (favouring longer answers) and self-enhancement (favouring its own answers). Panickssery et al. (2024) found that the better a model recognises its own text, the more it favours it. So pick a judge from a different model family than the model under test, and never let the tested model's own "passed" stand as evidence.
When answers can be compared in pairs, ask the judge which of two answers is better rather than asking it to score a single answer from 1 to 10. The MT-Bench paper notes that single-answer grading scales more easily, but it misses fine differences, produces more ties, and is less stable when the judge model changes. For a pairwise comparison, judge each pair twice with the order swapped, and count a win only when the same answer is preferred both times; otherwise record a tie.
Before trusting the judge, check it on about 30 examples you labelled by hand: the same answers, judged once by you and once by the model, then count how many agree. In that paper GPT-4 as judge agreed with human preferences more than 80% of the time, the same level as two humans agree with each other. If your task falls well short of that, fix the judging prompt and rubric, then check again. Check again too whenever the judging prompt, the reference material or the judge model version changes.
Use four inputs to locate the bottleneck
Hold the generator and answer budget fixed on the same questions:
These are diagnostic conditions, not a guaranteed ordering of scores. If correct evidence still fails, inspect the question, instructions, and generator. If it succeeds but real retrieval fails, inspect recall, reranking, and assembly along the retrieval pipeline.
Compare quality with cost and distribution
Use paired old-versus-new results on the same questions, examining regressions as well as improvements. Report sample size and sampling method; one or two changed answers in a small set do not demonstrate a stable gain. Repeated generation reveals randomness; resampling questions can estimate metric uncertainty but cannot repair unrepresentative coverage.
Track time to first token, total latency, retrieval and generation resources, and abstention on unanswerable cases. More candidates and context may improve recall while adding latency and distraction. Choose the system that fits the intended question distribution and budget rather than one isolated maximum score.
The Stanford ranked-retrieval evaluation chapter derives precision–recall curves, average precision, and discounted gain from ordered results.