Mathematics · advanced
Multivariable Optimization
Convexity, critical points, constrained optimization, and why SGD works on non-convex losses anyway.
Mental model
Convexity buys you one thing: no local minimum that is not also global. It does not promise a unique minimizer (a flat valley has many), nor that a minimum exists at all (e^-x descends forever), nor that gradient descent converges — too large a step still diverges, and an ill-conditioned Hessian still crawls. Most neural losses are non-convex yet empirically tractable. Constraints (Lagrange multipliers) turn 'find the best point' into 'find the best point on this boundary'.
How to study Multivariable Optimization
Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use Convex Optimization (Boyd & Vandenberghe), CS231n — Backpropagation & computational graphs, Neural Networks: Zero to Hero (Karpathy) to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.
Next, compare Multivariable Optimization with Gradient Descent, AdamW Optimizer, Linear Regression. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Condition number and learning rate and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.
Proof of understanding
- Explain the mechanism from first principles and identify the state it reads or changes.
- Give one situation where the concept is the right choice and one where it is not.
- Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
- Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.
Where it matters
Hyperparameter search, constrained resource allocation, training dynamics analysis.
Common mistakes
- Assuming every critical point is a minimum
- Ignoring learning rate when the Hessian is ill-conditioned
- Treating SGD convergence theory as guaranteeing global minima
Learn from primary sources
Practice and explain it back
Condition number and learning rate
Loss Hessian eigenvalues 1 and 100. Why one global lr=0.1 struggles? What helps?
Expected evidence: Ill-conditioned — oscillates along high-curvature dir; use Adam/normalization/smaller lr.
Open the interactive drill →Review prompts
- Convexity makes optimisation easier — but what exactly does it guarantee, and what does it not?
Build evidence
Use a roadmap capstone to turn this concept into working evidence.