Correlation & Hypothesis Testing

corr = df[["salary","experience"]].corr()

# Single pair:
r = df["salary"].corr(df["experience"])
print(f"Correlation: {r:.3f}")

# Full matrix:
df.select_dtypes("number").corr()

Pearson correlation r ranges from -1 to 1. r=1 perfect positive, r=-1 perfect negative, r=0 no linear relationship. Rule of thumb: |r| > 0.7 strong, 0.4-0.7 moderate, < 0.4 weak.

# Example: ice cream sales and drowning deaths are correlated
# because both increase in summer — not because ice cream causes drowning.

# Correlation: two variables move together
# Causation: one variable causes the other

# To establish causation you need:
# - Controlled experiments (A/B tests)
# - Randomised assignment
# - Ruling out confounding variables

This is one of the most common analyst interview questions. Correlation is easy to compute; causation requires experimental design. An analyst who confuses correlation with causation will give wrong business recommendations.

from scipy import stats

group_a = df[df["team"] == "A"]["sales"]
group_b = df[df["team"] == "B"]["sales"]

t_stat, p_value = stats.ttest_ind(group_a, group_b)

print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")

if p_value < 0.05:
    print("Statistically significant difference (reject H0)")
else:
    print("No significant difference (fail to reject H0)")

p-value < 0.05 means there is less than a 5% probability the difference is due to chance — by convention, this is "statistically significant". Used for A/B test analysis and comparing performance between segments.

# H0 (Null): No difference between groups
#   "Team A and Team B have the same average sales"

# H1 (Alternative): There is a difference
#   "Team A and Team B have different average sales"

# Reject H0 when p-value < significance level (0.05)
# Fail to reject H0 when p-value >= 0.05

The null hypothesis is always the "nothing interesting is happening" claim. You test whether the data provides enough evidence to reject it. You never "prove" H0 — you only reject it or fail to reject it.

# Pearson — linear relationship, assumes normal distribution:
df["salary"].corr(df["experience"])  # default

# Spearman — rank-based, works for non-linear and non-normal:
df["salary"].corr(df["experience"], method="spearman")

from scipy.stats import spearmanr
r, p = spearmanr(df["salary"], df["experience"])

Use Spearman when data is skewed or when you suspect a monotonic (but not necessarily linear) relationship. Salary and experience often have a Spearman correlation higher than Pearson because the relationship is not strictly linear.