Tutorials › Python › Data Cleaning with pandas

Data Cleaning with pandas

Fix messy real-world data — duplicates, wrong types, inconsistent values, and bad formats

Data cleaning takes 60–80% of a data analyst's time. Real datasets have duplicate rows, wrong data types, inconsistent category names ("Delhi" vs "delhi" vs "Delhi "), and columns that should be split or combined. Pandas has a comprehensive set of tools for all of these. This tutorial covers the cleaning operations you will use in every project.

Example

The standard data cleaning workflow
import pandas as pd

df = pd.read_csv("raw_data.csv")

# 1. First look
print(df.shape)         # rows, columns
print(df.info())        # types and null counts
print(df.duplicated().sum())  # how many duplicate rows?

# 2. Remove duplicates
df.drop_duplicates(inplace=True)
df.drop_duplicates(subset=["OrderID"], inplace=True)  # by key column

# 3. Fix data types
df["Salary"] = df["Salary"].astype(int)
df["OrderDate"] = pd.to_datetime(df["OrderDate"], dayfirst=True)
df["Amount"] = df["Amount"].str.replace("₹","").str.replace(",","").astype(float)

# 4. Standardise string columns
df["City"] = df["City"].str.strip().str.title()
df["Region"] = df["Region"].str.upper()

# 5. Rename columns
df.rename(columns={"Emp_Name": "Employee", "Dept": "Department"}, inplace=True)
df.columns = df.columns.str.strip().str.lower().str.replace(" ","_")

# 6. Drop unnecessary columns
df.drop(columns=["Unnamed: 0", "notes", "temp_col"], inplace=True)

Key Points

  • ✓df.duplicated().sum() counts duplicate rows; drop_duplicates() removes them
  • ✓df.columns.str.lower().str.replace(" ","_") standardises all column names at once
  • ✓pd.to_datetime() is essential for date columns — enables date filtering and time analysis
  • ✓inplace=True modifies the DataFrame directly — without it, you must reassign: df = df.drop_duplicates()
  • ✓str.strip().str.title() is the standard fix for city/name columns with mixed case and spaces

Practice Question

A column "City" has values "delhi", "Delhi", "DELHI", "delhi " (with trailing space). Which operation standardises all of these to "Delhi"?