Mathematics · core
A/B Testing for Engineers
Sample size, statistical power, practical significance, SRM checks, and multiple-comparison traps in product experiments.
Mental model
An A/B test is a hypothesis test with a deployment budget. Power asks whether you'd detect a real effect; practical significance asks whether the effect matters; SRM catches broken randomization before you trust the p-value.
How to study A/B Testing for Engineers
Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use Evan Miller — Sample Size Calculator, Microsoft ExP platform paper, Spotify — New Experimentation Platform, Part 2 (validity checks & SRM) to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.
Next, compare A/B Testing for Engineers with Search Evals, LLM Evals, Product Analytics, Model Evaluation. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Design an A/B test with power, Interpret a sample ratio mismatch and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.
Proof of understanding
- Explain the mechanism from first principles and identify the state it reads or changes.
- Give one situation where the concept is the right choice and one where it is not.
- Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
- Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.
Where it matters
Feature flags, model rollouts, ranking changes, pricing experiments.
Common mistakes
- Peeking at results and stopping early when p < 0.05
- Ignoring sample ratio mismatch (SRM)
- Shipping a statistically significant but tiny lift
- Running dozens of metrics without correction
Learn from primary sources
Practice and explain it back
Design an A/B test with power
You want to test a new ranking model on search CTR (baseline 8%, hope for +0.5pp). Define: primary metric, MDE, α, target power, estimated n per arm, duration given 50k daily users split 50/50, peeking policy, and guardrail metrics.
Expected evidence: A one-page test plan with n≈7k+ per arm (similar proportion formula), ~3–4 days runtime, no peeking until fixed n, guardrails on latency and null-result rate.
Open the interactive drill →Interpret a sample ratio mismatch
Planned 50/50 split; after 24h you observe 52% control / 48% treatment (n=100k total). χ² test flags SRM at p=0.001. What do you do before trusting the CTR lift?
Expected evidence: Stop the test readout; investigate bucketing bugs, bot traffic, locale skew, or deployment issues. Do not ship on a broken randomization.
Open the interactive drill →Review prompts
- Why is peeking at A/B results mid-test dangerous?
- Statistical significance vs statistical power?
Build evidence
A/B test analysis report
Analyze a mock or real experiment: check SRM, compute CI on the lift, interpret p-value and practical significance, and write a ship/no-ship recommendation.
- Document sample sizes per variant and check for sample ratio mismatch
- Report point estimate and 95% CI for the primary metric
- State null/alternative, p-value interpretation, and Type I/II tradeoff in plain English
- Recommend ship, iterate, or extend the test with a one-paragraph rationale