advanced practice · coding problem
Pre-LayerNorm residual block
x=[1,2], sublayer(x)=x+[0.1,−0.1]. Write pre-norm update: x + sublayer(LayerNorm(x)). If LayerNorm is identity here, new x?
The mechanism
A transformer block is two sublayers — multi-head attention then a position-wise MLP — each wrapped in a residual connection and layer norm. Residuals keep gradients flowing; pre-LayerNorm (norm before the sublayer) makes deep stacks train stably.
Expected evidence
[1.1,1.9] — residual preserves a gradient highway.
Try this drill in the practice workspace →
Hints when you need them
- Pre-norm: normalize before sublayer.
- Output = x + f(norm(x)).
Watch for these mistakes
- Dropping residual connections and watching deep models fail to train
- Post-LayerNorm placement causing instability in deep stacks
- Mismatched dimensions between the attention output and the MLP