core practice · coding problem
Perplexity vs classification accuracy
LM benchmark: PPX dropped 5% but exact-match QA accuracy flat. What does that imply about the eval?
The mechanism
Offline eval (on a held-out test set) tells you the model got better. Online eval (A/B test on real users) tells you the product got better. You need both — neither alone is enough. Scope: this card owns the offline/online split and the classical failure modes — held-out sets, baselines, leakage, distribution shift. Building a grader for free-text output is `llm-evals`; comparing whole systems on cost and latency is `quality-cost-latency-measurement`.
Expected evidence
LM loss improved token distribution but task metric insensitive or needs few-shot/prompt — check task-specific eval.
Try this drill in the practice workspace →
Hints when you need them
- PPX measures generative calibration.
- Downstream tasks need their own harness.