Search & IR · core
TF-IDF
Term frequency × inverse document frequency weighting for relevance.
Mental model
TF-IDF says a term matters more when it appears often in a document (TF) but is rare across the corpus (IDF). It is the conceptual ancestor of BM25.
How to study TF-IDF
Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use Introduction to Information Retrieval (Manning et al.) — §6.2.2 Tf-idf weighting to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.
Next, compare TF-IDF with BM25. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Calculate a TF-IDF score by hand and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.
Proof of understanding
- Explain the mechanism from first principles and identify the state it reads or changes.
- Give one situation where the concept is the right choice and one where it is not.
- Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
- Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.
Where it matters
Baseline ranking, feature input to learning-to-rank models.
Common mistakes
- Ignoring document length so long docs always win
- Using raw counts instead of dampened term frequency
Learn from primary sources
Practice and explain it back
Calculate a TF-IDF score by hand
Given term counts per document and document frequencies, compute the TF-IDF weight of a term in a document. Use dampened TF (1 + log tf) and IDF = log(N / df).
Expected evidence: A single numeric weight, higher for frequent-here-but-rare-overall terms.
Open the interactive drill →Review prompts
- What does the IDF factor accomplish in TF-IDF?
Build evidence
Use a roadmap capstone to turn this concept into working evidence.