Evaluating Retrieval and Generation: From Evidence to Correct Answers
Fluent answers do not prove correct retrieval, and an existing citation does not necessarily support its neighboring claim. Evaluate finding evidence separately from answering with it, then test their combination. The Ragas paper distinguishes retrieved context, answer faithfulness, and generation quality, and explores model-assisted scoring.
Define an evaluation case
For a fictional equipment-manual assistant, store the question, applicable manual version, supporting passage locations, required conditions, and expected behavior when evidence is absent. Answers can have multiple valid phrasings; evidence and essential facts are more stable than one reference sentence.
Include exact identifiers, paraphrases, multi-passage questions, conflicts between old and current versions, and unanswerable questions. Draw from intended use. Questions generated by one model directly from documents may echo the source vocabulary and underestimate real users' phrasing. Generated additions still need sampled checks of answerability and evidence.
Keep near-duplicates in the same split so that tuning questions do not return as paraphrased test cases. Freeze the corpus snapshot: a previously unanswerable question becomes a different test when a new document answers it.
Measure recall and ranking separately
If a question has 3 labeled relevant passages and 2 occur in the first 5 results, Recall@5 = 2/3. Relevant passages may duplicate one another or provide separate necessary facts. Also track whether the evidence contains all facts needed to solve the question, rather than rewarding repeated content.
MRR measures the position of the first relevant result. First hits at positions 1, 4, and no hit give (1 + 1/4 + 0) / 3 ≈ 0.417. It suits finding one useful result quickly but cannot establish completeness for multi-evidence questions. nDCG supports graded relevance and rewards valuable early results, while still depending on labels.
rankings = [["a", "b", "c"], ["x", "y", "z"], ["m", "n", "o"]]
relevant = [{"a", "c"}, {"z"}, {"p"}]
recalls = [len(set(r) & truth) / len(truth)
for r, truth in zip(rankings, relevant)]
print(round(sum(recalls) / len(recalls), 3)) # 0.667
This computes macro-average Recall@3 for three invented questions, weighting each equally. With incomplete relevance judgments, describe recall over labeled relevant items; do not treat every unlabeled passage as irrelevant.
Distinguish faithful, correct, and complete
Faithfulness asks whether supplied context supports the claims. Correctness asks whether claims match applicable facts. Completeness asks whether necessary conditions are included. An answer can faithfully repeat an obsolete manual and still be wrong for the current version. One correct but irrelevant sentence is not a complete answer.
Check both citation precision—does the cited passage support the claim?—and evidence coverage—are claims needing support actually sourced? A long list of links establishes neither. On unanswerable questions, separately count unsupported answers and appropriate abstention.
Model judges can screen at scale, but prompts, reference material, and model versions affect their scores. Reevaluate reordered or rephrased equivalent answers to test stability, and manually inspect disagreements and borderline cases. The tested model's own declaration of success is not evidence of success.
Use four inputs to locate the bottleneck
Hold the generator and answer budget fixed on the same questions:
These are diagnostic conditions, not a guaranteed ordering of scores. If correct evidence still fails, inspect the question, instructions, and generator. If it succeeds but real retrieval fails, inspect recall, reranking, and assembly along the retrieval pipeline.
Compare quality with cost and distribution
Use paired old-versus-new results on the same questions, examining regressions as well as improvements. Report sample size and sampling method; one or two changed answers in a small set do not demonstrate a stable gain. Repeated generation reveals randomness; resampling questions can estimate metric uncertainty but cannot repair unrepresentative coverage.
Track time to first token, total latency, retrieval and generation resources, and abstention on unanswerable cases. More candidates and context may improve recall while adding latency and distraction. Choose the system that fits the intended question distribution and budget rather than one isolated maximum score.
The Stanford ranked-retrieval evaluation chapter derives precision–recall curves, average precision, and discounted gain from ordered results. Follow how one relevant or irrelevant result changes the curve, then apply the calculation to a small labeled ranking from your own evaluation set.