Data Splits and Leakage

The 2008 KDD Cup asked entrants to detect breast cancer from mammography data. Looking back at that competition, Kaufman et al. (2011) note that the patient ID, a feature most competitors ignored, had tremendous and unexpected predictive power. The data had been compiled from several clinical-study, institution, or equipment sources; each source received a consecutive block of IDs, and the merge did not obscure the source. The ID therefore revealed where a patient came from, and some sources had a high cancer rate.

That signal came from how the data were assembled, not from the breast images; screening real new patients involves no such block of IDs sorted by source, so the competition scores did not carry over. This is data leakage: training, feature construction, model selection, or evaluation uses information that would not legitimately be available in the target prediction setting, making evaluation results overly optimistic.

Dropping the ID does not necessarily clean the data. Kaufman et al. point out that if ID ranges correspond to health care facilities in different places with different equipment, other traces may remain: if, say, grey levels are higher at the location with the high cancer rate, a model without the ID can still recognize the source from the remaining data, and its score stays optimistic.

The same error is common in research. Kapoor and Narayanan's 2022 survey of fields that adopted machine-learning methods found 17 fields where leakage errors had been identified, collectively affecting 329 papers, some with wildly overoptimistic conclusions.

The defense is to state what information exists at the moment of prediction, then make the split and every preprocessing step respect that boundary. The example near the end, a prompt revised again and again, applies the rules below to an LLM evaluation.

Define the Information Boundary

Before splitting the data, define the following metadata:

unit             prediction object, such as a patient, order, or device
prediction_time  when the output must be made
horizon          target time span, such as the next 30 days
available_info   information genuinely available and allowed then
label_time       when the label occurs, matures, and may be revised
split_key        entities, groups, and times that must not cross sets

Legitimacy is relative to the deployment information set $\mathcal{I}(t)$. A field existing in a database does not mean it had been generated at prediction time $t$, or that business rules permit the model to use it.

Leakage Types

Type	Example	Distortion	Control
target/post-outcome	billing finalized after discharge predicts admission-time risk	the outcome produces the feature, reversing the causal order	audit event times and lineage
temporal	random time-series split or future revisions	training sees the future regime	forward split and data vintages
entity/group	one patient/device appears on both sides	identity or history is memorized	group, sometimes group+time split
duplicate	different crops of one image or templated documents cross sets	test is not a new case	cluster or group near-duplicates before splitting
preprocessing	scaler, imputer, PCA, selection fit globally	test distribution changes transforms	fit all transforms inside every training fold; only transform test data
model selection	test repeatedly changes model/threshold	test becomes validation	lock test or obtain a new holdout
benchmark contamination	training contains questions and answers	measures reproduction	provenance, deduplication, source/time isolation

Note: A strongly correlated proxy feature is not necessarily illegitimate; what matters is whether the information actually exists at prediction time and is permitted for use. A proxy legitimately available at deployment may still be a shortcut, a source of bias, or a fragile correlation. That is a robustness problem rather than leakage in the strict sense. Assess it with subgroup analysis, intervention tests, and distribution-shift checks rather than treating every strongly correlated feature as leakage.

Split to Simulate Use

Different deployment questions require different splitting rules:

Deployment question	Preferred split	Does not answer
new independent same-regime cases	stratified random	temporal change or identity memory
new patients/users/devices	group	future policy or environmental change
future period	temporal forward	new institution or region
new institution/region	external/group holdout	every external environment
spatially dependent cases	blocked spatial	arbitrary unseen-region extrapolation

Tip: Stratification preserves label proportions; it does not prevent the same entity, nearby time points, or duplicates from crossing the training/test boundary.

Open full-size image

First separate the test set on the right. On the left, each row uses one blue fold for validation and the green folds for fitting. Fit preprocessing again inside each training fold. The test set stays outside model selection and is used only for final evaluation. Five folds demonstrate the procedure here; how many folds to use, and how to group them, follows from the deployment question in the previous section.

scikit-learn developers, Cross-validation: evaluating estimator performance. · BSD 3-Clause

Pipeline Invariant

For training indices $I_{tr}$ and validation indices $I_{va}$, every learned transform $T_\eta$ must satisfy

$$\hat\eta=\operatorname{fit}(T,D_{I_{tr}}), \qquad X'_{va}=T_{\hat\eta}(X_{I_{va}}).$$

This covers scaling, imputation, vocabulary construction, PCA, feature selection, target encoding, oversampling, and learned data augmentation.

• Cross-validation (CV): Refit every transform inside each training fold.
• Nested CV: Use the outer loop to evaluate generalization and the inner loop for hyperparameter or feature selection.
• Common mistake: Selecting features on all the data before running CV still causes leakage.

A Tiny Train-Only Transformation

The scikit-learn leakage guide uses the same rule: learn preprocessing on training data, then reuse it unchanged. Suppose training feature values are 2 and 4, while validation contains 100. Training-only centering learns 3, so validation becomes 97. Global centering would learn $106/3$ and let validation change the representation of the training rows.

train = [("A", 2.0), ("B", 4.0)]
valid = [("C", 100.0)]
assert {g for g, _ in train}.isdisjoint(g for g, _ in valid)
center = sum(x for _, x in train) / len(train)
assert [x - center for _, x in train] == [-1.0, 1.0]
assert [x - center for _, x in valid] == [97.0]

This toy split tests new entities: A and B are in training, C only in validation. If the task is to predict the next visit of an already known patient, the model can legitimately use the history available at that time; a group-disjoint test answers a different question, how the model does on patients it has never seen. For a future-new-patient test, choose a time cutoff: train only on earlier rows whose labels have matured by that cutoff, and test only on later rows from patients absent from training. This leaves out of the test the returning patients who had visited before the cutoff, so the test population changes; the report should state how many were excluded.

A pipeline prevents cross-fold fitting mistakes, not every within-training target leak. Target encoding can let a row's own label enter its feature; use out-of-fold or appropriately time-ordered training encodings. Resampling changes training data only, never validation/test prevalence.

Worked Example: Prompt Iteration Turns the Test Set into a Validation Set

Setting: Imagine you are tuning the system prompt of an assistant that answers questions only from your own product manual. You have 30 questions with written reference answers. After each prompt revision you run all 30 and ship the version with the highest score.

Flawed design:

• Every prompt revision is guided by the score on these 30 questions. After twenty rounds they have become a validation set, and part of the gain is fitting to those 30 questions themselves.
• Few-shot examples are picked from the same 30 questions, which writes test questions and answers into the prompt.
• To get more questions, a model first rewrites each one five ways, and the test set is then split at random, so rewrites of one question land in both the tuning set and the test set.
• The retrieval-augmented questions are written from manual chunks. Chunks of the same document produce some tuning questions and some test questions, so both sides share the same context and wording.

Stronger design:

1. Split first, then expand: Group by original question (retrieval-augmented questions by source document), split into tuning and test sets, then do rewrites and augmentation inside the tuning set only.
2. Take examples from the tuning set only: Few-shot examples and sample answers in the prompt come only from the tuning set.
3. Tune on the tuning set only: Compare every prompt revision on the tuning set alone and keep the test set locked.
4. Test once: Run the test once on the final version and report results by question type. If you change the prompt again afterwards, use a fresh batch of questions as the test set.
5. Fix the run settings: Record the model ID, temperature and grading method on the run card and keep them unchanged before and after the test.

If the score drops clearly on fresh questions, part of the old score came from fitting those 30 questions. It is the same kind of problem as the patient ID in KDD Cup 2008: the score absorbed information that real use would not provide.

Leakage Probes

Use the following probes and negative controls to look actively for leakage:

• Time audit: Record event, ingestion, and revision times for every feature.
• Single-feature probe: Train on each suspicious feature alone; trace its lineage if performance is unusually high.
• ID probe: Remove or shuffle entity identifiers and high-cardinality proxy features. The KDD Cup 2008 patient ID is exactly what this probe is meant to catch.
• Duplicate audit: Check exact and near duplicates.
• Label permutation: If the full pipeline still performs significantly above a chance baseline after labels are randomly shuffled, suspect a bug or leakage.
• Boundary assertions: Assert that groups do not cross sets, training times precede test times for forward-in-time evaluation, and transforms are never fitted on test data.
• Fresh holdout: Obtain new held-out data for benchmarks that have been inspected repeatedly.

Limitation: Permutation tests do not find every leak: they only catch leakage that the pipeline recomputes from the shuffled labels. A leaky feature precomputed from the true labels and stored in a table stops matching once the labels are shuffled, so the shuffled score falls to chance and the test looks clean while the leak remains. Combine controls with lineage.

Distinctions

• Overfitting: The model learns noise in the training sample; this can happen even with a valid pipeline. High training accuracy alone is not leakage: an overfit model also scores high on its training data.
• Leakage: The development process crosses an information boundary.
• Distribution shift: The deployment distribution differs from the evaluation distribution.
• Confounding: A common cause distorts a causal interpretation.

All four can coexist: after a leak is fixed, overfitting and distribution shift may still be there.

Pre-Release Checklist

prediction_time: explicit
label_horizon_and_maturity: explicit
unit_and_group_key: explicit
split_algorithm_and_seed: versioned
all_learned_transforms_fit_on_train_only: true
test_used_for_selection: false
duplicate_and_overlap_audit: complete
feature_availability_lineage: reviewed
known_residual_risks: listed

My reading