Skip to main content

Loading and Inspecting DataFrames

Loading is a schema decision, not merely a call to read_csv.

import pandas as pd

orders = pd.read_csv(
    "orders.csv",
    usecols=["order_id", "customer_id", "created_at", "amount"],
    dtype={"order_id": "string", "customer_id": "string"},
    parse_dates=["created_at"],
    na_values=["", "NA", "null"],
)

Ingestion checklist​

  1. Identify delimiter, encoding, headers, decimal convention, and missing sentinels.
  2. Select required columns and specify semantic dtypes where inference is risky.
  3. Parse dates deliberately; localize or convert time zones explicitly afterward.
  4. Inspect shape, head, info, duplicate keys, and missingness.
  5. Assert the invariants the next stage relies on.
assert orders["order_id"].notna().all()
assert orders["order_id"].is_unique
assert orders["amount"].ge(0).all()

Index choice​

Keep the default range index unless a domain key genuinely benefits selection or alignment. A key can remain an ordinary column:

orders = orders.set_index("order_id", verify_integrity=True)
orders = orders.reset_index()

Do not use a non-unique business field as an index merely because it looks like an identifier.

Rename at the boundary​

Normalize names once, close to ingestion:

orders = orders.rename(columns=str.strip)
orders.columns = orders.columns.str.lower().str.replace(" ", "_", regex=False)

Preserve a mapping when external names must remain traceable.

Scale boundary​

Use chunksize when independent chunks can be aggregated incrementally. If the workflow requires repeated full-table joins or shuffles beyond memory, changing the execution engine is usually better than elaborate chunk bookkeeping.

Source​