Missing Values

A missing balance does not tell us whether an account has no money or whether its balance was never observed. The reason for absence determines whether filling is justified. Pandas provides a common API over representations including pd.NA, NaN, and NaT.

Detect

import pandas as pd

df = pd.DataFrame({
    "required_key": [1, 2, 3], "country": ["CA", None, "FR"],
    "account_id": ["A", "A", "B"], "time": [1, 2, 1],
    "balance": pd.Series([10, pd.NA, pd.NA], dtype="Int64"),
}).sort_values(["account_id", "time"])
missing_by_column = df.isna().sum()
complete_rows = df.notna().all(axis="columns")

Before filling, missing_by_column reports two missing balances and one missing country; the other columns have none. complete_rows is true only for the first row.

Use isna and notna; equality with a missing sentinel is not a reliable test. Inspect both counts and rates, and split them by meaningful cohorts when aggregate missingness may hide a systematic pattern.

Choose a policy

Action	Appropriate when
Preserve	absence itself is meaningful or later logic handles it
Drop	the row/column is unusable and deletion bias is acceptable
Fill a constant	a domain-specific sentinel has explicit meaning
Forward/back fill	ordering is valid and values persist across adjacent observations
Interpolate	a defensible model connects neighboring points
Model/impute	uncertainty and leakage are controlled explicitly

df = df.dropna(subset=["required_key"])
df["country"] = df["country"].fillna("unknown")
df["balance"] = df.groupby("account_id")["balance"].ffill()

The balances now read as follows:

Account	Time	Before	After `ffill`
A	1	10	10
A	2	`<NA>`	10
B	1	`<NA>`	`<NA>`

A’s second row inherits 10 from A’s earlier observation. B has no earlier balance in its own group, so it stays missing: A’s 10 cannot cross the account boundary. All required keys are present, so dropna removes no rows; the missing country becomes "unknown". dropna and fillna return results by default; the assignments explicitly update df.

Sort by the relevant entity and time keys before directional filling. Never fill across group boundaries accidentally.

Dtypes and reductions

Nullable extension dtypes such as Int64, boolean, and string can preserve semantic types while representing missing data. Specify them at important boundaries rather than depending on inference.

Many reductions skip missing values by default. Set skipna or required counts (min_count) explicitly when that default changes the meaning of the result.

Unknown is not zero or false

The missing-data rules also affect logic: pd.NA == pd.NA is unknown, and bool(pd.NA) raises TypeError. Empty strings and infinity are not missing by default. An all-missing sum is zero unless a minimum valid count is required. Nullable comparisons followed by .all() can skip unknown values, so validate presence separately when a column is required.

empty = pd.Series([pd.NA, pd.NA], dtype="Int64")
assert empty.sum() == 0
assert pd.isna(empty.sum(min_count=1))
assert empty.ge(0).all()
assert not empty.notna().all()

Record the decision

A cleaning pipeline should state which columns may be missing, why, and how each policy affects downstream analysis. Keep an indicator column when imputation may itself carry predictive or diagnostic information.

Source

Explore connections

Referenced by (1)

Pandas

Detect​

Choose a policy​

Dtypes and reductions​

Unknown is not zero or false​

Record the decision​

Source​