Skip to main content

Evaluation and evidence

Before you rely on a model, an agent or a RAG system, test it on your own task and know what a score can show. Vendor benchmark claims are read with Reading Model Cards, Licenses and Benchmarks; dataset provenance and distribution shift are on the foundations shelf.

Reading Order​

StepArticleYour job
1Evaluating a Model on Your Own TasksEvaluate on your own task: replay real work against one exact configuration and a threshold set in advance.
2Data Splits and LeakageKeep test cases out of everything the system learned from or was tuned on.
3Evaluating Retrieval and Generation: From Evidence to Correct AnswersMeasure retrieval, ranking, faithfulness and answer quality separately to find where a RAG system fails.
4Probability Calibration, Decision Thresholds, and AbstentionTurn scores into actions: calibrate probabilities, set a cost-based threshold and decide when to abstain.
5Judging Evidence in AI ExperimentsJudge other people’s claims: what a specification, a vendor statement, a benchmark or an observation can prove.
Explore connectionsOpen network