Handling Nulls & Outliers

import numpy as np
import pandas as pd

np.nan   # float NaN — NumPy missing
None     # Python object null
pd.NA    # Pandas nullable integer/string NA

# In practice — all treated as missing by Pandas:
pd.isnull(np.nan)  # True
pd.isnull(None)    # True
pd.isnull(pd.NA)   # True

Pandas unifies all three with isnull()/isna(). The distinction matters when you mix Python objects and NumPy arrays. For most analyst work: they all mean "missing" — treat them the same.

Q1 = df["salary"].quantile(0.25)
Q3 = df["salary"].quantile(0.75)
IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

outliers = df[(df["salary"] < lower) | (df["salary"] > upper)]
print(f"{len(outliers)} outliers detected")

The 1.5*IQR rule (Tukey method) is the standard. It is what a box plot uses to mark outliers. Always inspect outliers before removing — they may be valid extreme values, not data errors.

from scipy import stats

z_scores = stats.zscore(df["salary"].dropna())
outliers = df[abs(z_scores) > 3]

# Manual Z-score:
df["z"] = (df["salary"] - df["salary"].mean()) / df["salary"].std()
outliers = df[df["z"].abs() > 3]

Z-score > 3 means the value is more than 3 standard deviations from the mean — statistically rare. Works well for normally distributed data. IQR is better for skewed distributions like salary or revenue.

lower = df["salary"].quantile(0.01)  # 1st percentile
upper = df["salary"].quantile(0.99)  # 99th percentile

df["salary_capped"] = df["salary"].clip(lower=lower, upper=upper)

Clipping (Winsorization) is often better than removal — you keep all rows but limit extreme values. Use it when outliers are data entry errors but you cannot afford to lose the rows.

# Fill salary null with the mean salary of that department:
df["salary"] = df.groupby("department")["salary"].transform(
    lambda x: x.fillna(x.mean())
)

Group-level imputation is far more accurate than global mean imputation. A missing salary in Engineering should be filled with the Engineering mean, not the company-wide mean. transform() applies the function within each group and returns the same-length result.