core practice · coding problem
Interpret a sample ratio mismatch
Planned 50/50 split; after 24h you observe 52% control / 48% treatment (n=100k total). χ² test flags SRM at p=0.001. What do you do before trusting the CTR lift?
The mechanism
An A/B test is a hypothesis test with a deployment budget. Power asks whether you'd detect a real effect; practical significance asks whether the effect matters; SRM catches broken randomization before you trust the p-value.
Expected evidence
Stop the test readout; investigate bucketing bugs, bot traffic, locale skew, or deployment issues. Do not ship on a broken randomization.
Try this drill in the practice workspace →
Hints when you need them
- SRM means the assignment mechanism is wrong.
- Effect estimates are biased under SRM.
Watch for these mistakes
- Peeking at results and stopping early when p < 0.05
- Ignoring sample ratio mismatch (SRM)
- Shipping a statistically significant but tiny lift
- Running dozens of metrics without correction