core practice · coding problem
Estimate training token budget
Corpus: 1M docs × 800 tokens each. 3 epochs. Total tokens seen? If Chinchilla says ~20 tokens/param for 1B model, rough token budget?
The mechanism
In ML, the dataset is the product. The biggest quality wins usually come from cleaning, deduping, and filtering the data — not from changing the model. Pipelines need versioning and provenance the same way code does.
Expected evidence
2.4B tokens for corpus; 20B tokens suggested for 1B params.
Try this drill in the practice workspace →
Hints when you need them
- tokens = docs × tokens/doc × epochs.
- Scale data with model size for compute-optimal training.