A/B Testing Deep Dive for Data Analysts
From hypothesis to decision — sample size, p-values, power, Type I/II errors, multiple testing corrections, novelty effects, and how to run experiments that produce trustworthy results.
A/B testing is the gold standard for establishing causality in product decisions. Every other analytical technique — cohort analysis, funnel analysis, regression — can identify correlations. Only a well-run experiment can tell you whether a change caused an outcome. That distinction matters enormously when a product team is deciding whether to ship a new feature or roll back a redesign.
The mechanics of running an A/B test are straightforward. The statistics are where most teams go wrong — p-hacking, underpowered experiments, peeking at results early, and running so many tests simultaneously that some will appear significant by chance. This chapter gives you the complete mental model: what the numbers actually mean, what can go wrong, and how to interpret and communicate results responsibly.
The A/B Testing Framework
Writing a Good Hypothesis
A vague hypothesis ("we think this will improve conversion") produces uninterpretable results. A precise hypothesis specifies the change, the expected direction and magnitude, and the mechanism — so if the result surprises you, you learn something either way.
Sample Size and Statistical Power
The three inputs you must define before running any experiment:
import scipy.stats as stats
import numpy as np
def ab_sample_size(baseline_rate, mde, alpha=0.05, power=0.80):
"""
Calculate required sample size per variant for a two-proportion z-test.
baseline_rate : current conversion rate, e.g. 0.42
mde : minimum detectable effect (absolute), e.g. 0.08
"""
treatment_rate = baseline_rate + mde
effect_size = abs(
(treatment_rate - baseline_rate) /
np.sqrt((baseline_rate * (1 - baseline_rate) +
treatment_rate * (1 - treatment_rate)) / 2)
)
analysis = stats.power.tt_ind_solve_power(
effect_size=effect_size,
alpha=alpha,
power=power,
alternative='two-sided'
)
return int(np.ceil(analysis))
# Example: onboarding test
baseline = 0.42 # 42% current completion
mde = 0.08 # we want to detect ≥ 8pp improvement
n_per_variant = ab_sample_size(baseline, mde)
print(f"Required per variant: {n_per_variant:,}")
print(f"Total users needed: {n_per_variant * 2:,}")
# If daily traffic = 800 users, split 50/50 → 400 per variant per day
days_needed = n_per_variant / 400
print(f"Test duration: {days_needed:.1f} days")Understanding p-values (and What They Don't Mean)
The p-value is the probability of observing a difference at least as large as the one you measured, assuming the null hypothesis (no real difference) is true. A p-value of 0.03 means: "if there were truly no difference between control and treatment, we would see a gap this large or larger in only 3% of experiments by random chance."
Statistical vs practical significance
A result can be statistically significant but practically useless. With 2 million users in each variant, you can detect a 0.01% lift with p < 0.05 — but a 0.01% improvement in conversion probably doesn't justify the engineering and maintenance cost of a new feature. Always report effect size alongside the p-value.
# Full A/B test analysis
from scipy import stats
import numpy as np
# Results after test duration
control_visitors = 4200
control_converts = 1764 # 42.0%
treatment_visitors = 4200
treatment_converts = 1974 # 47.0%
# Proportions
p_control = control_converts / control_visitors
p_treatment = treatment_converts / treatment_visitors
# Two-proportion z-test
count = np.array([treatment_converts, control_converts])
nobs = np.array([treatment_visitors, control_visitors])
z_stat, p_value = stats.proportions_ztest(count, nobs)
# Effect size: absolute and relative lift
abs_lift = p_treatment - p_control
rel_lift = abs_lift / p_control
# Confidence interval for the difference
se = np.sqrt(p_control*(1-p_control)/control_visitors +
p_treatment*(1-p_treatment)/treatment_visitors)
ci_low = abs_lift - 1.96 * se
ci_high = abs_lift + 1.96 * se
print(f"Control: {p_control:.1%}")
print(f"Treatment: {p_treatment:.1%}")
print(f"Absolute lift: {abs_lift:+.1%} ({ci_low:+.1%} to {ci_high:+.1%}, 95% CI)")
print(f"Relative lift: {rel_lift:+.1%}")
print(f"p-value: {p_value:.4f}")
print(f"Significant: {'Yes' if p_value < 0.05 else 'No'}")Type I and Type II Errors
| H₀ is TRUE (no real effect) | H₀ is FALSE (real effect exists) | |
|---|---|---|
| Test says: SIGNIFICANT | Type I Error (False Positive) You ship a feature that doesn't work. Probability = α (0.05). | Correct Decision ✓ You detected a real improvement. Probability = Power (0.80). |
| Test says: NOT SIGNIFICANT | Correct Decision ✓ You correctly didn't ship. Probability = 1−α (0.95). | Type II Error (False Negative) You missed a real improvement. Probability = β (0.20). |
Multiple Testing Problem and Bonferroni Correction
If you test 20 independent hypotheses simultaneously with α = 0.05, you expect one false positive by chance — even if none of the variants actually work. This is the multiple testing problem. It is why running 10 simultaneous A/B tests and declaring the "winning" one is statistically unreliable.
# Bonferroni correction: divide alpha by number of tests
# Conservative but simple
n_tests = 5 # running 5 simultaneous variants
alpha_orig = 0.05
alpha_bonf = alpha_orig / n_tests # = 0.01
print(f"Original α: {alpha_orig}")
print(f"Bonferroni-corrected α: {alpha_bonf}")
print("Each individual test must achieve p < 0.01 to be considered significant")
# Benjamini-Hochberg (FDR): less conservative than Bonferroni
from statsmodels.stats.multitest import multipletests
p_values = [0.003, 0.041, 0.072, 0.009, 0.118] # from 5 tests
reject, p_corrected, _, _ = multipletests(p_values, alpha=0.05, method='fdr_bh')
for i, (p, p_c, r) in enumerate(zip(p_values, p_corrected, reject)):
print(f"Test {i+1}: raw p={p:.3f} corrected p={p_c:.3f} reject={r}")Common A/B Testing Mistakes
Communicating A/B Test Results to Stakeholders
A test result is not a decision. The analyst's job is to present the evidence clearly so the decision-maker can weigh the statistical signal against business context, implementation cost, and strategic priority.