# BM25

> The standard lexical ranking function: TF saturation + IDF + length normalization.

Canonical: https://learn.significanthobbies.com/curriculum/concepts/bm25

The standard lexical ranking function: TF saturation + IDF + length normalization.

- Difficulty: core
- Tracks: Search & IR

## Mental model

BM25 fixes TF-IDF's two flaws: term frequency saturates (the 10th occurrence adds little via the k1 parameter) and document length is normalized (the b parameter). It is still the strongest single keyword ranker and the lexical half of hybrid search.

## Where it matters

Default scorer in Elasticsearch, OpenSearch, Lucene, Tantivy; lexical leg of hybrid retrieval.

## Common mistakes

- Believing embeddings make BM25 obsolete — they are complementary
- Leaving k1/b at defaults without tuning on your corpus
- Skipping document length normalization
- Not building search evals to compare against vector retrieval

## Primary sources

- [Okapi BM25 (Wikipedia)](https://en.wikipedia.org/wiki/Okapi_BM25) (doc)
- [Elastic — Practical BM25 Part 2: The BM25 Algorithm and its Variables](https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables) (article)

## Practice

### Calculate a BM25 score by hand

Given tf, df, N, document length, average document length, k1=1.2 and b=0.75, compute the BM25 score of a query term for a document.

**Expected evidence:** A numeric score; verify TF saturation by checking the 10th occurrence adds little.

### Implement a BM25 ranker

On top of your inverted index, implement BM25 scoring and return the top-k documents for a multi-term query.

**Expected evidence:** Ranked top-k results for a query, scored by summed per-term BM25.

## Review prompts

- Why does BM25 normalize by document length?
- How is BM25 different from TF-IDF?
- When can BM25 outperform embedding-based retrieval?
- How would you combine BM25 and vector search?

## Build evidence

- **Implement BM25 search in HighSignal** — Index a corpus of articles and serve ranked keyword search using BM25.
- **Hybrid search v0 in HighSignal** — Combine BM25 and vector retrieval with reciprocal rank fusion.

## Prerequisites

- [Inverted Index](https://learn.significanthobbies.com/curriculum/concepts/inverted-index)
- [TF-IDF](https://learn.significanthobbies.com/curriculum/concepts/tf-idf)

## Related concepts

- [Hybrid Search](https://learn.significanthobbies.com/curriculum/concepts/hybrid-search)
- [Search Evals](https://learn.significanthobbies.com/curriculum/concepts/search-evals)
- [Reranking](https://learn.significanthobbies.com/curriculum/concepts/reranking)

## Learning paths

- [9-Day Reset](https://learn.significanthobbies.com/curriculum/roadmaps/reset-9-day)
- [30-Day Retrieval Basics](https://learn.significanthobbies.com/curriculum/roadmaps/retrieval-30-day)
- [90-Day AI Search & Infrastructure](https://learn.significanthobbies.com/curriculum/roadmaps/ai-search-infra-90-day)
- [12-Month Advanced AI Infrastructure](https://learn.significanthobbies.com/curriculum/roadmaps/ai-infra-12-month)
- [Disk-First Databases & RAM](https://learn.significanthobbies.com/curriculum/roadmaps/db-disk-first)
- [HLD Practice](https://learn.significanthobbies.com/curriculum/roadmaps/hld-practice)