Mathematics · core
Linear Regression
Fitting a line (or hyperplane) by least squares; residuals, R², and the geometry of projection.
Mental model
Linear regression finds the best linear predictor — geometrically, it projects the target vector onto the column space of your features. Least squares minimizes squared error, which is why outliers pull the line hard.
How to study Linear Regression
Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use An Introduction to Statistical Learning — full text (PDF), Stanford CS229 — Main lecture notes (linear regression, least squares), MIT 18.06 L16 — Projection matrices and least squares (Strang) to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.
Next, compare Linear Regression with Derivatives & Gradients, Gradient Descent, Estimation & Confidence Intervals. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Fit a simple linear regression and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.
Proof of understanding
- Explain the mechanism from first principles and identify the state it reads or changes.
- Give one situation where the concept is the right choice and one where it is not.
- Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
- Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.
Where it matters
Forecasting, feature importance, baseline models before deep learning.
Common mistakes
- Extrapolating far outside the training range
- Treating high R² as proof of causation
- Ignoring multicollinearity among features
Learn from primary sources
Practice and explain it back
Fit a simple linear regression
Points: (1, 2), (2, 3), (3, 5). Fit y = a + bx by least squares. Compute a, b, and the predicted ŷ at x = 4.
Expected evidence: b ≈ 1.5, a ≈ 0.17, ŷ(4) ≈ 6.17. (Exact: b = 3/2, a = 1/6.)
Open the interactive drill →Review prompts
- Why is least-squares regression called a projection?
Build evidence
Least-squares regression from scratch
Fit y = Xβ via normal equations (XᵀX)β = Xᵀy using your matmul — no sklearn. Report β, residuals, R².
- Matches sklearn/polyfit on a 2-feature toy dataset within 1e-6
- Plot residuals; comment on one outlier's leverage