Bias, Variance & Overfitting

Mental model

Bias is systematic error from too-simple a model; variance is sensitivity to which sample you drew. Overfitting is low bias + high variance. The gym is holdout validation, not training R².

How to study Bias, Variance & Overfitting

Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use StatQuest — Bias and Variance to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.

Next, compare Bias, Variance & Overfitting with Maximum Likelihood Estimation, Multivariable Optimization, Momentum Backtest Discipline. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Diagnose overfit with a train/holdout split and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.

Proof of understanding

• Explain the mechanism from first principles and identify the state it reads or changes.
• Give one situation where the concept is the right choice and one where it is not.
• Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
• Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.

Where it matters

ML training, backtest overfitting, feature discipline.

Common mistakes

• Tuning on the test set
• Chasing training loss without validation
• Equating complex with better

Learn from primary sources

• StatQuest — Bias and Variance (video)

Practice and explain it back

Review prompts

• Explain the bias-variance tradeoff.

Build evidence

Learning paths

• 30-Day Probability & Statistics
• 12-Week Active Math Stack