Tutorials › Python › Outlier Detection in Python

Outlier Detection in Python

Find and handle extreme values that distort your analysis using IQR and Z-score

Outliers are data points that are far from the rest of the data. They can be genuine extreme values (a CEO salary in an employee dataset) or data errors (a typo: 750000 instead of 75000). Not handling outliers leads to skewed means, misleading charts, and inaccurate models. Two common methods: IQR (Interquartile Range) — robust, non-parametric; Z-score — assumes approximately normal distribution.

Example

Detect outliers with IQR and Z-score
import pandas as pd
import numpy as np
from scipy import stats

df = pd.read_csv("salaries.csv")

# METHOD 1 — IQR (recommended for skewed data)
Q1 = df["Salary"].quantile(0.25)
Q3 = df["Salary"].quantile(0.75)
IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

outliers_iqr = df[(df["Salary"] < lower) | (df["Salary"] > upper)]
print(f"IQR outliers: {len(outliers_iqr)}")
print(f"  Lower fence: {lower:,.0f}")
print(f"  Upper fence: {upper:,.0f}")

# METHOD 2 — Z-score (for roughly normal distributions)
df["zscore"] = np.abs(stats.zscore(df["Salary"].dropna()))
outliers_z = df[df["zscore"] > 3]  # beyond 3 standard deviations
print(f"Z-score outliers: {len(outliers_z)}")

# Visualise with box plot
import matplotlib.pyplot as plt
plt.figure(figsize=(8, 4))
plt.boxplot(df["Salary"].dropna(), vert=False)
plt.title("Salary Distribution — Box Plot (outliers as dots)")
plt.show()

Key Points

  • ✓IQR method: outliers are < Q1 - 1.5×IQR or > Q3 + 1.5×IQR — same as box plot whiskers
  • ✓Z-score: values with |z| > 3 are usually outliers (beyond 3 standard deviations)
  • ✓Always investigate before removing — some outliers are real data, not errors
  • ✓Options: remove, cap (winsorize), or log-transform the column
  • ✓Box plot visually shows outliers as individual dots beyond the whiskers

Practice Question

A salary column has Q1=40000, Q3=80000. What is the IQR upper fence for outlier detection?