Model Evaluation Under Distribution Shift

Evaluation estimates how a system behaves under one target distribution, threshold, workflow, and error cost; it does not assign a permanent score.

For deployment distribution $Q(x,y)$ and loss $\ell$:

$$R_Q(f)=\mathbb{E}_{(x,y)\sim Q}[\ell(f(x),y)].$$

A test average estimates this risk only when test approximates $Q$, remains outside development, and uses the right labels and loss. A narrow iid interval does not include unknown future shift.

Specify the Object

system           model + preprocessing + threshold + human process
unit             one prediction object
population       people/time/place to which results extend
prediction_time  information available then
outcome_horizon  label window and maturity
action           what the prediction triggers
loss             false-positive, false-negative, delay, and abstention cost

If an action changes later data, static test evaluates the predeployment policy, not the feedback loop.

Shift Types

Type	Change	Example	Boundary
covariate	$p(x)$	device, region, season	feature drift does not prove $p(y\mid x)$ stable
label/prior	$p(y)$	disease or fraud prevalence	needs labels or assumptions
concept/conditional	$p(y\mid x)$	policy changes relationships	hard to detect from unlabeled inputs
measurement	recording process	sensor or coding change	may resemble real behavior change
selection	inclusion mechanism	only reviewed cases are labeled	test represents the selected population

They can overlap. Input drift does not guarantee performance loss, and no detected drift does not guarantee safety.

Splits Answer Different Questions

Question	Design	Main limitation
new independent same-regime cases	random holdout/CV	not time, region, or entity shift
new entities	group holdout	few groups yield wide uncertainty
future performance	rolling/forward test	covers only observed historical changes
new institution	external validation	one site is not all sites
unbiased estimate after selection	nested CV or locked test	still depends on sampling mechanism

Repeatedly viewing test, publishing only the best seed, or tuning to an external benchmark converts test into validation.

Metrics Answer Different Questions

Accuracy follows class prevalence. Precision/recall depend on threshold and prevalence. ROC-AUC measures ranking, not an operating point. PR-AUC emphasizes positives but has a prevalence-dependent baseline. Log loss and Brier score assess probability quality differently. Calibration is distribution- and sample-dependent.

For regression, MAE and RMSE weight tails differently; percentage errors can fail near zero; quantile loss targets conditional quantiles. Ranking metrics describe a ranking protocol, not downstream utility. Select metrics from actions and error costs, not leaderboard convention.

Worked Example: 1% Fraud

Among 10,000 transactions, 100 are fraud. Always predicting normal gives 99% accuracy and zero recall. Suppose a threshold flags 200 transactions, 60 truly fraudulent:

precision = 60 / 200 = 30%
recall    = 60 / 100 = 60%

Usefulness also depends on 200 review costs, the 40 missed cases, customer friction, delayed labels, and next month's prevalence. Select the threshold on validation, then evaluate it on a later test; selecting and reporting it on the same test is optimistic.

Three Uncertainty Layers

1. Evaluation sample: bootstrap or analytic intervals; resample clusters/blocks when group or time dependence exists.
2. Training randomness: initialization, order, and nondeterministic kernels; repeat seeds.
3. Environment: future policy, behavior, sensors, and population; iid intervals usually omit it.

Report denominators, slice counts, failed runs, and which uncertainty an interval represents.

Slices, Thresholds, and Abstention

Overall metrics may hide failures by group, device, language, time, or difficulty. Predefine risk-relevant slices; many post-hoc slices create selection effects and need confirmation. High-consequence systems may abstain or route to humans, but reviewer skill, queue capacity, and selection bias become part of the system. Confidence is not automatically calibrated probability, especially under shift.

Baselines and Stress Tests

Use constants, seasonal rules, business processes, and simple models; remove suspicious features; stress time, noise, missingness, length, language, or device; test new cohorts/sites; exercise invalid input and refusal paths; and compare architectures at controlled parameter, data, and training budgets.

intended_decision: ...
target_population_and_period: ...
prediction_time_and_horizon: ...
data_version_and_split: ...
baselines: [...]
selection_metric: ...
final_metrics_and_thresholds: ...
uncertainty: sample, seed, environment
slices_and_minimum_counts: [...]
leakage_and_contamination_checks: [...]
resource_and_latency_budget: ...
known_failures_and_out_of_scope: [...]
monitoring_and_revalidation_trigger: ...

Public benchmarks support comparison but invite overfitting to fixed tasks, languages, hardware, and success metrics. Internal tests better match use but may be small and unaudited. Model Cards and the NIST AI RMF provide reporting questions; they do not certify safety or fairness.

This note concerns supervised prediction. Causal effects, A/B tests, RL policies, open-ended generation, and human–AI collaboration require different estimands and designs.