Data and evaluation

Evaluation starts with the decision a result will support. Data provenance, prediction time and the cost of mistakes determine the split and metrics before a model is selected.

Start with dataset suitability and leakage-resistant splitting, then distribution shift. Calibration connects probabilities to thresholds and abstention. Local-model task replay and retrieval/generation evaluation bring these ideas into full systems; the evidence note explains how far a comparison supports a claim.

Reading Order

Step	Article	What it explains
1	Datasets: Provenance, Leakage, and Evaluation	A map from data-generating processes and provenance to leakage-resistant splits, distribution-shift evaluation, and authoritative public sources.
2	Data Splits and Leakage	Block target, group, temporal, and preprocessing leakage by defining prediction time, entities, and train-only pipelines.
3	Model Evaluation Under Distribution Shift	Start from deployment distribution, splits, metrics, thresholds, and uncertainty instead of treating one test score as universal ability.
4	Probability Calibration, Decision Thresholds, and Abstention	Separate prediction accuracy, reliable probabilities, and worthwhile actions using independent calibration and evaluation data.
5	Evaluating Local Models and Agent Tasks	A reproducible protocol for measuring whether one exact local model configuration clears a real workflow threshold.
6	Evaluating Retrieval and Generation: From Evidence to Correct Answers	Measure evidence retrieval, ranking, faithfulness, and answer quality separately to locate a RAG system’s bottleneck.
7	Judging Evidence in AI Experiments	A practical way to label specifications, vendor claims, benchmark results, observations, and recommendations without pretending they prove the same thing.

Use What You Read

Separate training, selection, calibration and final testing. Report the population and operating conditions along with a score, and preserve failures that would change the deployment decision.

Return to the AI reading paths to choose a neighboring topic.

My reading · Opened notes are remembered in this browser.