Datasets: Provenance, Leakage, and Evaluation

A downloadable table is not automatically evidence. A dataset records one collection regime, measurement process, version, and scope. Both missingness and inclusion can carry mechanisms.

Reading Order

1. Use this page to assess fitness.
2. Fix Data Splits and Leakage before learned processing.
3. Use Evaluation Under Distribution Shift to bind test results to a deployment distribution and cost.

Fitness Review

Dimension	Ask	Common failure
provenance	who produced it, why, and how?	a mirror loses definitions and version
unit	does a row mean person, event, window, or aggregate?	repeated entities treated as independent
population	whom should conclusions cover?	convenience sample called representative
measurement	how are variables and labels observed and revised?	administrative proxy treated as the construct
time	when are features and labels available?	future fields or revised values leak
missingness	who is absent and why?	imputation hides systematic invisibility
joins	are keys unique and at one granularity?	many-to-many join multiplies rows
rights	do license, consent, and privacy permit use?	public visibility treated as permission
reproducibility	can version, query, checksum, and transforms be rebuilt?	URL points only to changing latest data

Rejecting an unsuitable dataset early is a successful result.

Minimal Datasheet

name: dataset-name
version_or_vintage: immutable identifier
producer: responsible institution
purpose: original collection purpose
unit: one row means ...
population_and_scope: geography, time, inclusion
measurement: feature and label construction
known_exclusions: missing or censored groups
license_and_sensitivity: terms, consent, privacy
retrieval: url/query/date/checksum
transformations: ordered executable steps
split_keys: entity, group, time
intended_use: supported decisions
out_of_scope: prohibited or unsupported uses

Datasheets for Datasets offers a fuller framework. This minimal record supports traceability; it does not certify ethics or statistical validity.

Authoritative Starting Points

Domain	Source	Use	Still inspect
U.S. public data	Data.gov	cross-agency discovery	producing agency and terms
U.S. population/economy	Census	surveys and demographic/business data	sampling, weights, geography, versions
global development	World Bank	country indicators	definition changes and missingness
macro/finance	FRED	sourced time series	vintages, revisions, release time
global health	WHO Data	health indicators	reporting regimes and country coverage

An official producer is better for definitions and revisions, not automatically unbiased. Use community catalogs for discovery; return to producers when conclusions depend on definitions, licenses, or vintages.

Do Not Conflate

• Data quality: are values accurate, complete, and consistent?
• Leakage: did development cross the deployment information boundary?
• Distribution shift: does use differ from train/test?

Clean data can leak. Leakage-free data can fail after policy change. Imputation does not repair split design, and split design does not guarantee external validity.

Do not train a complex model while prediction time, target, entity boundaries, label maturity, rights, split key, baseline, or error costs remain undefined. Record unknowns as blockers rather than hiding them behind a default random split.

My reading · Opened notes are remembered in this browser.