advanced practice · coding problem
Compute cross-entropy in bits
True distribution P=(0.5, 0.5); model Q=(0.9, 0.1). Compute H(P) and cross-entropy H(P,Q) in bits (log base 2). Which term dominates?
The mechanism
Entropy measures average surprise of a distribution. Cross-entropy compares your predicted distribution to the true one — it's the natural classification loss. KL divergence is asymmetric: how many extra nats you pay when you use Q instead of P.
Expected evidence
H(P)=1 bit. H(P,Q) = −0.5·log₂0.9 − 0.5·log₂0.1 ≈ 0.08 + 1.66 ≈ 1.74 bits. The 1.66 dominates: the model gave 0.1 to a class that occurs half the time, and being confidently wrong is expensive.
Try this drill in the practice workspace →
Hints when you need them
- H(P)=−Σ p log₂ p.
- Cross-entropy uses Q as the model distribution.
Watch for these mistakes
- Treating KL as a metric (it's not symmetric)
- Using log base inconsistently (bits vs nats)
- Confusing entropy of data with entropy of a model