intro practice · implementation task
Build a search tokenizer
Write a tokenizer that lowercases, splits on non-alphanumerics, removes a small stop-word set, and applies simple suffix stemming. Use the exact same function for indexing and querying.
The mechanism
Tokenization decides what counts as a 'word' for search. The same analyzer must run at index time and query time, or the query term will never match the indexed term.
Expected evidence
tokenize('The Quick Brown Foxes!') → ['quick','brown','fox']Try this drill in the practice workspace →
Hints when you need them
- The analyzer must be symmetric across index and query time.
- Keep stop words minimal — they hurt short queries.
Watch for these mistakes
- Using different analyzers for indexing and querying
- Stemming too aggressively so distinct words collide
- Dropping stop words that carry meaning in short queries