Mathematics · core
Hypothesis Testing
Null and alternative hypotheses, p-values, significance, power, and Type I/II errors.
Mental model
Hypothesis testing asks: 'how surprising is this data if nothing real changed?' A p-value is not the probability the null is true — it's the probability of seeing data this extreme under the null. Low p → reject the null, but 'not significant' is not proof of no effect.
How to study Hypothesis Testing
Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use Seeing Theory — Hypothesis Testing, StatQuest — p-values to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.
Next, compare Hypothesis Testing with Linear Regression, LLM Evals. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Interpret a p-value correctly, Type I vs Type II errors and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.
Proof of understanding
- Explain the mechanism from first principles and identify the state it reads or changes.
- Give one situation where the concept is the right choice and one where it is not.
- Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
- Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.
Where it matters
Experiment analysis, clinical trials, model comparison gates.
Common mistakes
- Treating p < 0.05 as proof the effect is real
- Running many tests without correction (p-hacking)
- Confusing statistical significance with practical significance
Learn from primary sources
- Seeing Theory — Hypothesis Testing (article)
- StatQuest — p-values (video)
Practice and explain it back
Interpret a p-value correctly
An A/B test reports p = 0.03 for H₀: 'no difference in conversion'. The lift is 0.1% absolute (0.5% → 0.6%). Write two sentences: one correct interpretation of the p-value, one reason you might still not ship.
Expected evidence: Correct: if there were truly no effect, we'd see data this extreme ~3% of the time. Might not ship: effect is tiny (practical insignificance) or the test was one of many without correction.
Open the interactive drill →Type I vs Type II errors
A security patch test: H₀ = 'no change in error rate'. You fail to reject H₀, but errors actually rose 0.5%. (a) Which error type? (b) If α=0.05 and power=0.8, what does each number mean in one sentence?
Expected evidence: (a) Type II error (false negative) — missed a real regression. α=5% false-alarm rate if H₀ true; power=80% chance to detect a real effect of the assumed size.
Open the interactive drill →Review prompts
- What is a p-value, and what is it not?
- Define Type I and Type II errors.
Build evidence
A/B test analysis report
Analyze a mock or real experiment: check SRM, compute CI on the lift, interpret p-value and practical significance, and write a ship/no-ship recommendation.
- Document sample sizes per variant and check for sample ratio mismatch
- Report point estimate and 95% CI for the primary metric
- State null/alternative, p-value interpretation, and Type I/II tradeoff in plain English
- Recommend ship, iterate, or extend the test with a one-paragraph rationale
Analyze a fake trading signal
Given (or generate) a spurious 'alpha' signal on synthetic or historical noise, show why it fails: multiple testing, in-sample overfit, or non-stationarity.
- Reproduce the 'great' in-sample metric
- Show out-of-sample or holdout collapse
- Name the statistical mistake (p-hacking, data snooping, regime shift, etc.)
Simple momentum backtest vs SPY/QQQ
12-month momentum rule on 1–3 names; compare cumulative return to SPY or QQQ over the same window. State assumptions and survivorship bias.
- Rule written before seeing full-sample results (or use holdout half)
- Equity curve: strategy vs benchmark
- Honest limitations section (costs, slippage, overfitting)