advanced practice · coding problem
Condition number and learning rate
Loss Hessian eigenvalues 1 and 100. Why one global lr=0.1 struggles? What helps?
The mechanism
Convexity buys you one thing: no local minimum that is not also global. It does not promise a unique minimizer (a flat valley has many), nor that a minimum exists at all (e^-x descends forever), nor that gradient descent converges — too large a step still diverges, and an ill-conditioned Hessian still crawls. Most neural losses are non-convex yet empirically tractable. Constraints (Lagrange multipliers) turn 'find the best point' into 'find the best point on this boundary'.
Expected evidence
Ill-conditioned — oscillates along high-curvature dir; use Adam/normalization/smaller lr.
Try this drill in the practice workspace →
Hints when you need them
- Curvature varies by direction.
- Adaptive per-parameter methods help.
Watch for these mistakes
- Assuming every critical point is a minimum
- Ignoring learning rate when the Hessian is ill-conditioned
- Treating SGD convergence theory as guaranteeing global minima