| 1 | Evaluating a Model on Your Own Tasks | Evaluate on your own task: replay real work against one exact configuration and a threshold set in advance. |
| 2 | Data Splits and Leakage | Keep test cases out of everything the system learned from or was tuned on. |
| 3 | Evaluating Retrieval and Generation: From Evidence to Correct Answers | Measure retrieval, ranking, faithfulness and answer quality separately to find where a RAG system fails. |
| 4 | Probability Calibration, Decision Thresholds, and Abstention | Turn scores into actions: calibrate probabilities, set a cost-based threshold and decide when to abstain. |
| 5 | Judging Evidence in AI Experiments | Judge other people’s claims: what a specification, a vendor statement, a benchmark or an observation can prove. |