core practice · coding problem
AdamW vs L2-on-gradient
Weight w=2, grad=0.5, lr=0.1, decay=0.01. Compare one step: (a) L2 in gradient w−lr·(grad+decay·w) vs (b) AdamW-style w−lr·grad then w·(1−lr·decay).
The mechanism
AdamW is the standard optimizer for training transformers. It adapts the learning rate per parameter (Adam) and applies weight decay separately from the gradient (the "W"). The default choice unless you have a reason to switch.
Expected evidence
Both ≈ 1.948 (L2 1.94800, decoupled 1.94805). One step barely separates them — the point is that decoupled decay never enters the gradient, so it is not rescaled by the adaptive step.
Try this drill in the practice workspace →
Hints when you need them
- Coupled L2 adds decay·w to the gradient.
- AdamW applies decay directly on weights.