Mathematics · core
Estimation & Confidence Intervals
Point estimates, standard error, confidence intervals, and what '95% confident' actually means.
Mental model
A confidence interval is a procedure, not a probability statement about one interval. If you repeated the experiment many times, 95% of the intervals you'd construct would contain the true parameter. Wider interval = more uncertainty or less data.
How to study Estimation & Confidence Intervals
Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use MIT 18.650 Statistics for Applications to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.
Next, compare Estimation & Confidence Intervals with Hypothesis Testing, Linear Regression. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Construct a 95% confidence interval and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.
Proof of understanding
- Explain the mechanism from first principles and identify the state it reads or changes.
- Give one situation where the concept is the right choice and one where it is not.
- Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
- Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.
Where it matters
A/B test reporting, polling margins of error, model metric uncertainty.
Common mistakes
- Saying 'there is a 95% chance the true value is in this interval'
- Using the normal approximation with tiny samples
- Ignoring that CI width grows with desired confidence level
Learn from primary sources
Practice and explain it back
Construct a 95% confidence interval
n = 100 coin flips, 55 heads. Assuming a normal approximation, construct a 95% CI for the true heads probability p. (Use z = 1.96.)
Expected evidence: p̂ = 0.55, SE ≈ 0.0498, CI ≈ [0.452, 0.648].
Open the interactive drill →Review prompts
- What does a 95% confidence interval mean?
Build evidence
A/B test analysis report
Analyze a mock or real experiment: check SRM, compute CI on the lift, interpret p-value and practical significance, and write a ship/no-ship recommendation.
- Document sample sizes per variant and check for sample ratio mismatch
- Report point estimate and 95% CI for the primary metric
- State null/alternative, p-value interpretation, and Type I/II tradeoff in plain English
- Recommend ship, iterate, or extend the test with a one-paragraph rationale