core practice · implementation task
Build a search eval harness
Create a labelled query set (query → relevant doc ids) and a harness that scores any retriever with recall@k, MRR, and nDCG.
The mechanism
You cannot improve retrieval you cannot measure. A search eval is a fixed set of queries with known-good results plus a harness that scores any retriever — it turns ranking changes from vibes into numbers.
Expected evidence
A single command that prints the three metrics for a retriever.
Try this drill in the practice workspace →
Hints when you need them
- Freeze the query set before tuning.
- Score retrievers through one common interface.
Watch for these mistakes
- Tuning ranking without a frozen eval set
- Letting the eval set leak into tuning decisions
- Too few queries to detect real regressions