advanced practice · coding problem
Overfit a single batch sanity check
Training loss stuck high on 1 batch of 8 examples. Name two data bugs to check before changing the architecture.
The mechanism
A training loop is forward pass → loss → backward pass → optimizer step, repeated over batches of data. When training looks broken, the data is the usual culprit (leaked, mislabeled, wrong shape) — check it before blaming the model.
Expected evidence
Wrong labels/shapes, train-eval leakage, broken tokenizer, masked tokens all padding, etc.
Try this drill in the practice workspace →
Hints when you need them
- Karpathy recipe: overfit one batch first.
- If you cannot overfit one batch, data or loss is wrong.
Check the sources
- Stanford CS336 — L2 PyTorch & resource accounting (Spring 2025)
- CS336 Assignment 2 — Systems (profile, Triton, distributed training)
- A Recipe for Training Neural Networks (Karpathy)
- The Illustrated GPT-2 — training a language model
- Let's reproduce GPT-2 (124M) — Karpathy
- Stanford CS336 — L9–11 Scaling laws (Spring 2025)
- How to Scale Your Model (JAX ML scaling book)