core practice · system design prompt
Design an A/B test with power
You want to test a new ranking model on search CTR (baseline 8%, hope for +0.5pp). Define: primary metric, MDE, α, target power, estimated n per arm, duration given 50k daily users split 50/50, peeking policy, and guardrail metrics.
The mechanism
An A/B test is a hypothesis test with a deployment budget. Power asks whether you'd detect a real effect; practical significance asks whether the effect matters; SRM catches broken randomization before you trust the p-value.
Expected evidence
A one-page test plan with n≈7k+ per arm (similar proportion formula), ~3–4 days runtime, no peeking until fixed n, guardrails on latency and null-result rate.
Try this drill in the practice workspace →
Hints when you need them
- One primary metric; everything else is diagnostic.
- Pre-register stopping rule before launch.
Watch for these mistakes
- Peeking at results and stopping early when p < 0.05
- Ignoring sample ratio mismatch (SRM)
- Shipping a statistically significant but tiny lift
- Running dozens of metrics without correction