Missing Values
Missingness is a property of the data-generating process, not merely a value to
replace. Pandas provides a common API over several representations, including
pd.NA, NaN, and NaT.
Detect
missing_by_column = df.isna().sum()
complete_rows = df.notna().all(axis="columns")
Use isna and notna; equality with a missing sentinel is not a reliable test.
Inspect both counts and rates, and split them by meaningful cohorts when aggregate
missingness may hide a systematic pattern.
Choose a policy
| Action | Appropriate when |
|---|---|
| Preserve | absence itself is meaningful or later logic handles it |
| Drop | the row/column is unusable and deletion bias is acceptable |
| Fill a constant | a domain-specific sentinel has explicit meaning |
| Forward/back fill | ordering is valid and values persist across adjacent observations |
| Interpolate | a defensible model connects neighboring points |
| Model/impute | uncertainty and leakage are controlled explicitly |
df = df.dropna(subset=["required_key"])
df["country"] = df["country"].fillna("unknown")
df["balance"] = df.groupby("account_id")["balance"].ffill()
Sort by the relevant entity and time keys before directional filling. Never fill across group boundaries accidentally.
Dtypes and reductions
Nullable extension dtypes such as Int64, boolean, and string can preserve
semantic types while representing missing data. Specify them at important
boundaries rather than depending on inference.
Many reductions skip missing values by default. Set skipna or required counts
(min_count) explicitly when that default changes the meaning of the result.
Record the decision
A cleaning pipeline should state which columns may be missing, why, and how each policy affects downstream analysis. Keep an indicator column when imputation may itself carry predictive or diagnostic information.