Data Cleaning — The Core Skill

df.isnull().sum()                # null count per column
df.isnull().sum() / len(df) * 100  # null percentage
df.isnull().any()                # True/False per column
df[df["salary"].isnull()]        # rows where salary is null

Always compute null percentage, not just count. 5 nulls in 10 rows (50%) needs different treatment than 5 nulls in 10,000 rows (0.05%). Percentage guides your imputation strategy.

# Drop rows with any null:
df.dropna(inplace=True)

# Drop only if ALL columns are null:
df.dropna(how="all")

# Fill with a value:
df["salary"].fillna(0)
df["city"].fillna("Unknown")

# Fill with statistical measures:
df["salary"].fillna(df["salary"].mean())
df["salary"].fillna(df["salary"].median())

# Forward fill (time-series):
df["price"].fillna(method="ffill")

Use mean for normally distributed columns, median for skewed data (salaries, prices). Forward fill is standard for time-series gaps. Never just drop nulls without understanding why they exist.

df.duplicated().sum()          # count duplicates
df.duplicated(keep="first")    # mark first as not duplicate
df.drop_duplicates(inplace=True)  # remove all duplicates

# Duplicate based on specific columns:
df.drop_duplicates(subset=["customer_id"], keep="last")

subset= lets you define what "duplicate" means — same customer_id even if other columns differ. keep="last" retains the most recent record, useful for update logs.

# Convert to numeric (coerce turns errors to NaN):
df["salary"] = pd.to_numeric(df["salary"], errors="coerce")

# Convert to datetime:
df["date"] = pd.to_datetime(df["date"], format="%d/%m/%Y")

# Convert to category (for repeated strings):
df["city"] = df["city"].astype("category")

# Convert to string:
df["pincode"] = df["pincode"].astype(str)

errors="coerce" in to_numeric() is critical — it converts unparseable values to NaN instead of crashing. Always use format= in to_datetime() when the date format is non-standard.

df["city"] = df["city"].str.strip()          # remove whitespace
df["city"] = df["city"].str.lower()          # lowercase
df["city"] = df["city"].str.replace("  ", " ")  # double spaces

# Replace values:
df["city"] = df["city"].replace({"mumbai": "Mumbai", "delhi": "Delhi"})

# Regex replace:
df["phone"] = df["phone"].str.replace(r"[^0-9]", "", regex=True)

Text standardisation is the most common cleaning task. The replace dict approach is scalable — build a mapping dict and apply it. Regex replace is powerful for removing non-numeric characters from phone numbers or IDs.