core practice · coding problem
CLT and required sample size
A/B test on conversion rate: baseline 5%, MDE 1pp (to 6%). Rough rule: need ~16p(1−p)/δ² samples per arm for Normal approx (p=0.05, δ=0.01). Estimate n per arm. Why is n quadratic in 1/δ?
The mechanism
Your data is one draw from a bigger game. The CLT says sample means from many independent pieces look Normal even when the underlying data is not — which is why confidence intervals and z-tests work on proportions and averages.
Expected evidence
n ≈ 16×0.05×0.95/0.0001 ≈ 7,600 per arm. Halving MDE quadruples required n — detecting small lifts is expensive.
Try this drill in the practice workspace →
Hints when you need them
- Standard error of proportion shrinks as 1/√n.
- Smaller δ → need more data to separate signal from noise.
Watch for these mistakes
- Applying CLT with n=5 and calling it rigorous
- Treating the sample as the population
- Ignoring dependence across samples (e.g. time-series)