Datasets: Provenance, Leakage, and Evaluation
A downloadable table is not automatically evidence. A dataset records one collection regime, measurement process, version, and scope. Both missingness and inclusion can carry mechanisms.
Reading Order
- Use this page to assess fitness.
- Fix Data Splits and Leakage before learned processing.
- Use Evaluation Under Distribution Shift to bind test results to a deployment distribution and cost.
Fitness Review
| Dimension | Ask | Common failure |
|---|---|---|
| provenance | who produced it, why, and how? | a mirror loses definitions and version |
| unit | does a row mean person, event, window, or aggregate? | repeated entities treated as independent |
| population | whom should conclusions cover? | convenience sample called representative |
| measurement | how are variables and labels observed and revised? | administrative proxy treated as the construct |
| time | when are features and labels available? | future fields or revised values leak |
| missingness | who is absent and why? | imputation hides systematic invisibility |
| joins | are keys unique and at one granularity? | many-to-many join multiplies rows |
| rights | do license, consent, and privacy permit use? | public visibility treated as permission |
| reproducibility | can version, query, checksum, and transforms be rebuilt? | URL points only to changing latest data |
Rejecting an unsuitable dataset early is a successful result.
Minimal Datasheet
name: dataset-name
version_or_vintage: immutable identifier
producer: responsible institution
purpose: original collection purpose
unit: one row means ...
population_and_scope: geography, time, inclusion
measurement: feature and label construction
known_exclusions: missing or censored groups
license_and_sensitivity: terms, consent, privacy
retrieval: url/query/date/checksum
transformations: ordered executable steps
split_keys: entity, group, time
intended_use: supported decisions
out_of_scope: prohibited or unsupported uses
Datasheets for Datasets offers a fuller framework. This minimal record supports traceability; it does not certify ethics or statistical validity.
Authoritative Starting Points
| Domain | Source | Use | Still inspect |
|---|---|---|---|
| U.S. public data | Data.gov | cross-agency discovery | producing agency and terms |
| U.S. population/economy | Census | surveys and demographic/business data | sampling, weights, geography, versions |
| global development | World Bank | country indicators | definition changes and missingness |
| macro/finance | FRED | sourced time series | vintages, revisions, release time |
| global health | WHO Data | health indicators | reporting regimes and country coverage |
An official producer is better for definitions and revisions, not automatically unbiased. Use community catalogs for discovery; return to producers when conclusions depend on definitions, licenses, or vintages.
Do Not Conflate
- Data quality: are values accurate, complete, and consistent?
- Leakage: did development cross the deployment information boundary?
- Distribution shift: does use differ from train/test?
Clean data can leak. Leakage-free data can fail after policy change. Imputation does not repair split design, and split design does not guarantee external validity.
Do not train a complex model while prediction time, target, entity boundaries, label maturity, rights, split key, baseline, or error costs remain undefined. Record unknowns as blockers rather than hiding them behind a default random split.