intro practice · coding problem
One BPE merge step
Corpus tokenized as [l,o,w,e,r,l,o,w,e,s,t]. Count adjacent pairs; merge the most frequent pair once. What is the new sequence?
The mechanism
LLM tokenizers (BPE, SentencePiece) split text into small reusable pieces — often subwords — so the model can handle any input even if it has not seen the exact word. The model and the tokenizer are a pair; swapping one breaks the other.
Expected evidence
lo appears twice → merge lo → [lo,w,e,r,lo,w,e,s,t].
Try this drill in the practice workspace →
Hints when you need them
- Count bigram frequencies.
- Merge highest-count pair into one token.