advanced practice · coding problem
Scaled dot-product attention weights
Q=K=[[1,0],[0,1]], V=[[1,2],[3,4]], d_k=2. Compute softmax(QKᵀ/√d_k)·V for one query row.
The mechanism
Self-attention lets every token look at every other token and pull in what is relevant. Each token emits a query, a key, and a value; the query-key dot products (scaled, softmaxed) become weights over the values. Causal masking blocks a token from seeing the future.
Expected evidence
Scaled scores [0.71, 0] → weights ≈ 0.67/0.33 → output ≈ [1.7, 2.7].
Try this drill in the practice workspace →
Hints when you need them
- Scale before softmax.
- Apply weights to values.
Watch for these mistakes
- Forgetting the 1/sqrt(d_k) scaling, which destabilizes the softmax
- Omitting the causal mask in a decoder, leaking future tokens
- Confusing the query/key/value roles