advanced practice · coding problem
Split embedding across heads
d_model=512, h=8. What is head_dim? If you project Q to 512 dims total, how many dims per head?
The mechanism
Attention lets each token in a sentence look at every other token and weight how relevant each one is. "Multi-head" runs several of these attention passes in parallel so the model can track different kinds of relationships at once (e.g. grammar, references, position).
Expected evidence
head_dim = 512/8 = 64.
Try this drill in the practice workspace →
Hints when you need them
- head_dim = d_model / num_heads.
- Each head runs attention in its own subspace.