core practice · coding problem
LoRA parameter count
W is 4096×4096 (frozen). LoRA rank r=8. How many trainable params in A (4096×r) and B (r×4096)?
The mechanism
LoRA fine-tunes a big model cheaply by adding small trainable matrices alongside the frozen weights. You only update around 1% of the parameters yet get most of the quality of full fine-tuning, and you can swap adapters at inference time.
Expected evidence
4096·8 + 8·4096 = 65,536 trainable vs 16M full matrix.
Try this drill in the practice workspace →
Hints when you need them
- Train A and B only; W stays fixed.
- Params = d·r + r·d.