Model Evaluation Under Distribution Shift

In 2019 a group of researchers followed ImageNet's original procedure to collect a fresh set of test images. A wide range of image classifiers lost 11 to 14 points of accuracy on the new test set, yet their ranking stayed largely the same (Recht et al., 2019). No model changed; only the source of the test data did. The original test score described only the original data.

Open full-size image

Each blue point is one model: horizontal position is its original test accuracy, vertical position its accuracy on newly collected test data. Points below the dashed diagonal lost accuracy, even though the red fitted line shows a strong relationship between the two scores. The panels cover CIFAR-10 and ImageNet; error bars are 95% Clopper–Pearson intervals. A ranking that broadly persists can coexist with a systematic drop in absolute performance.

Recht, Roelofs, Schmidt & Shankar, ImageNetV2 project: original vs. new test accuracy.

Evaluation therefore estimates how a system behaves under one target distribution, threshold, workflow, and error cost; it does not assign a permanent score.

Call the data the model will actually face the deployment distribution $Q(x,y)$, where $x$ is the input and $y$ the true outcome; the loss $\ell$ measures how bad one prediction is. The target risk is

$$R_Q(f)=\mathbb{E}_{(x,y)\sim Q}[\ell(f(x),y)].$$

In plain words, the target risk is how wrong the model is, on average, on the data it will really be used on. A test average estimates this risk only when the test samples are approximately drawn from $Q$, the test set remains outside development, and the labels and loss are correctly defined. If the test samples come from the same source as the training data and are independent of one another, even a narrow confidence interval covers only that source and not future distribution shift: ImageNetV2's 11-to-14-point drop lies far outside the original test set's confidence interval.

Specify the Object

system           model + preprocessing + threshold + human process
unit             one prediction object
population       people/time/place to which results extend
prediction_time  information available then
outcome_horizon  label window and maturity
action           what the prediction triggers
loss             false-positive, false-negative, delay, and abstention cost

If an action changes later data, static test evaluates the predeployment policy, not the feedback loop.

Shift Types

Type	Change	Example	Boundary
covariate	$p(x)$	device, region, season	feature drift does not prove $p(y\mid x)$ stable
label/prior	$p(y)$	disease or fraud prevalence	needs labels or assumptions
concept/conditional	$p(y\mid x)$	policy changes relationships	hard to detect from unlabeled inputs
measurement	recording process	sensor, coding, or questionnaire change	may resemble real behavior change
selection	inclusion mechanism	only reviewed cases are labeled	test represents the selected population

They can overlap. Input drift does not guarantee performance loss, and no detected drift does not guarantee safety.

An individual developer most often meets three other shifts: their own users' prompts drift away from the set used in development; a new language or domain arrives; the provider changes the model behind the same name. For how to notice and handle the last one, see evaluating a model on your own tasks.

Splits Answer Different Questions

Question	Design	Main limitation
new independent same-regime cases	random holdout/CV	not time, region, or entity shift
new entities	group holdout	few groups yield wide uncertainty
future performance	rolling/forward test	covers only observed historical changes
new institution or region	external validation	one external site cannot represent every external environment
unbiased estimate after selection	nested CV or locked test	computationally expensive and still dependent on the sampling mechanism

Repeatedly viewing test, publishing only the best seed, or tuning to an external benchmark converts test into validation.

Metrics Answer Different Questions

Choose metrics from the actions and error costs, rather than copying a leaderboard's choice. The fraud example in the next section works through accuracy, precision and recall and shows how they can give opposite impressions; the other common metrics answer these questions.

For classification, ROC-AUC measures ranking and does not choose an operational threshold. PR-AUC emphasizes the positive class, but its baseline depends on the proportion of positives. Log loss and Brier score assess different aspects of probability quality. Calibration compares predicted probabilities with observed frequencies; it changes with the distribution and sample size.

Worked Example: 1% Fraud

Among 10,000 transactions, 100 are fraudulent. Always predicting “normal” gives 99% accuracy (the share of correct predictions) yet catches no fraud: recall is zero. Accuracy is dominated by the majority class, here the normal transactions.

Two other numbers are needed. Precision asks how many of the flagged transactions are really fraud; recall asks how many of the real frauds were flagged. Suppose a threshold flags 200 transactions, 60 truly fraudulent:

precision = 60 / 200 = 30%
recall    = 60 / 100 = 60%

Both numbers move with the threshold: lower it and more transactions are flagged, so recall rises and precision usually falls.

If nothing is flagged, precision has a zero denominator and is undefined; if a test slice contains no positives, recall is undefined. Report these cases rather than silently treating them as successful predictions.

Usefulness also depends on the cost of 200 manual reviews, losses from the 40 missed fraud cases, friction for legitimate customers, delayed labels, and next month's prevalence. Tuning the threshold to maximize utility on the test set and then reporting that same utility overestimates generalization. Select the threshold on validation, then evaluate it on a later test.

Prevalence changes precision too. Here 140 of the 200 flags are normal transactions, so the false-positive rate is $140/9900\approx1.414\%$. For prevalence $\pi$ (the share of all transactions that are fraud), true-positive rate $t$ (the share of frauds flagged, which is recall), and false-positive rate $f$ (the share of normal transactions flagged),

$$\operatorname{precision}=\frac{\pi t}{\pi t+(1-\pi)f},\qquad \operatorname{recall}=t.$$

The numerator counts flagged frauds and the denominator counts all flagged transactions. If only prevalence changes and the score distribution within each class stays the same (prior shift), $t$ and $f$ stay fixed. Raise prevalence to 5%: the numerator is $0.05\times0.6=0.03$, the denominator adds $0.95\times0.01414\approx0.0134$, and precision is about $0.03/0.0434\approx69.1\%$, while recall remains 60%. The same model at the same threshold goes from 30% to 69% precision only because fraud became more common. See the scikit-learn metric definitions.

How the costs of false alarms and misses set the threshold is worked through in probability calibration, decision thresholds, and abstention; a ROC-AUC value alone cannot choose it.

Three Uncertainty Layers

1. Evaluation sample: analytic or bootstrap intervals; preserve pairing when comparing models on the same cases, and resample clusters/blocks when group or time dependence exists.
2. Training randomness: initialization, order, and nondeterministic kernels; repeat seeds.
3. Environment: future policy, behavior, sensors, and population; intervals from same-source independent samples usually omit it.

Reporting only a mean ± an unspecified interval confuses these sources of uncertainty. Report numerators and denominators, sample counts for each slice, failed runs, and which uncertainty each interval represents.

Slices, Thresholds, and Abstention

Overall metrics may hide failures in important groups or slices defined by device, language, time, or difficulty. Define slices from the intended use and risks before seeing the results. Many post-hoc slices create selective findings that need correction or confirmation on new data.

Where errors have serious consequences, the model can abstain or route some cases to humans. Evaluation must also account for reviewer skill, processing capacity, queue capacity, and selection bias. Model confidence is not the probability of being correct; under shift, both probability calibration and uncertainty estimates can become unreliable.

For the biases of public benchmarks and of your own test set, see judging evidence in AI experiments.

My reading