Tutorials › Python › Handling Missing Values (NaN)

Handling Missing Values (NaN)

Detect, remove, and fill missing values — critical for accurate analysis

Missing values (NaN — Not a Number) are present in almost every real dataset. How you handle them affects every calculation. Ignoring nulls gives wrong counts. Summing columns with nulls gives wrong totals in some contexts. Choosing how to fill or drop nulls requires business judgment — not just a formula. Pandas represents missing values as NaN (for numbers) and NaT (for dates). Both are treated consistently by pandas functions.

Example

Detect and handle missing values
import pandas as pd
import numpy as np

df = pd.read_csv("data.csv")

# DETECT
df.isnull()              # True where NaN
df.isnull().sum()        # count nulls per column
df.isnull().sum() / len(df) * 100  # % null per column

# A column-level summary
null_report = pd.DataFrame({
    "null_count": df.isnull().sum(),
    "null_pct": (df.isnull().sum() / len(df) * 100).round(1)
}).sort_values("null_pct", ascending=False)

# DROP rows with any null
df.dropna(inplace=True)

# Drop only if specific columns have nulls
df.dropna(subset=["CustomerID", "OrderDate"], inplace=True)

# Drop columns with > 50% nulls
threshold = len(df) * 0.5
df.dropna(thresh=threshold, axis=1, inplace=True)

# FILL — choose the right method
df["Salary"].fillna(df["Salary"].median(), inplace=True)  # numeric: median
df["City"].fillna("Unknown", inplace=True)                # categorical: placeholder
df["OrderDate"].fillna(method="ffill", inplace=True)      # time series: forward fill
df["Score"].fillna(0, inplace=True)                       # counts: zero
💡 Never blindly drop all rows with nulls — that can remove most of your data. Always check null % first and decide per column.

Key Points

  • ✓df.isnull().sum() is the first null check — run it on every new dataset
  • ✓dropna() without parameters drops any row with even one null — often too aggressive
  • ✓fillna(median()) for numeric skewed data; fillna(mean()) for normally distributed
  • ✓fillna(method="ffill") fills with the previous valid value — for time-series data
  • ✓NaN arithmetic: any operation on NaN returns NaN — so one null in a column skews nothing (pandas skips NaN in sum/mean by default)

Practice Question

A "City" column has 30% null values. What is the most appropriate way to handle them?