EDA — Exploratory Data Analysis Workflow

# Step 1: Load and inspect
df = pd.read_csv("data.csv")
df.shape, df.dtypes, df.head()

# Step 2: Missing values
df.isnull().sum() / len(df) * 100

# Step 3: Descriptive statistics
df.describe()

# Step 4: Distribution of numeric columns
df.hist(figsize=(14, 8), bins=20)

# Step 5: Categorical value counts
for col in df.select_dtypes("object").columns:
    print(df[col].value_counts())

# Step 6: Correlation matrix
df.corr()

# Step 7: Outlier detection
# (IQR or Z-score per numeric column)

EDA has no single right answer, but this sequence covers the essentials. The goal is to understand distributions, relationships, and data quality issues before any analysis begins.

# Numeric summary:
df.describe()

# Include all types:
df.describe(include="all")

# Only categorical:
df.describe(include="object")

# Select by dtype:
num_cols = df.select_dtypes(include="number").columns
cat_cols = df.select_dtypes(include="object").columns

select_dtypes() is essential for applying operations only to numeric or categorical columns without hardcoding names. This makes your EDA code reusable across different datasets.

num_df = df.select_dtypes(include="number")
num_df.hist(
    figsize=(14, 10),
    bins=20,
    color="#1d4ed8",
    edgecolor="white",
    layout=(3, 4)
)
plt.suptitle("Distributions of Numeric Columns", y=1.02, fontsize=14)
plt.tight_layout()
plt.show()

.hist() on a DataFrame plots all numeric columns in a grid automatically. layout=(rows, cols) controls the grid shape. This one command replaces writing individual hist() calls for each column.

# Group statistics:
df.groupby("department")["salary"].describe()

# Visualise:
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
sns.boxplot(data=df, x="department", y="salary", ax=axes[0])
sns.barplot(data=df, x="department", y="salary", ax=axes[1])
plt.tight_layout()

Box plot shows the full distribution; bar plot shows just the mean. Show both when presenting to stakeholders — the box plot reveals variability the bar chart hides.

# Correlation matrix:
corr = df.select_dtypes("number").corr()

# Find highly correlated pairs:
high_corr = (corr.abs() > 0.7) & (corr != 1.0)
high_pairs = [(i, j) for i in corr.columns for j in corr.columns if high_corr.loc[i,j] and i < j]
print("High correlation pairs:", high_pairs)

# Heatmap:
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm", center=0)

High correlation (>0.7) between two features means one may be redundant. In regression analysis, multicollinearity (high correlation between predictors) inflates standard errors and makes coefficients unreliable.