Search & IR · advanced
Click Models & Position Bias
Why clicks are not relevance labels, and how to debias implicit feedback.
Mental model
A click means the user examined the result AND judged it worth clicking, so the observed rate confounds relevance with position — rank 1 gets clicked far more than rank 5 for identical content. Training on raw clicks therefore teaches the model to reproduce the ranking that generated them, a feedback loop that entrenches whatever was already on top. Inverse-propensity weighting breaks it by dividing each click by the estimated probability that the position was examined, which is why deliberate result randomisation is worth its cost.
How to study Click Models & Position Bias
Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use the linked roadmap and primary implementation references to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.
Next, compare Click Models & Position Bias with Learning to Rank, A/B Testing for Engineers. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Debias clicks by inverse propensity and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.
Proof of understanding
- Explain the mechanism from first principles and identify the state it reads or changes.
- Give one situation where the concept is the right choice and one where it is not.
- Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
- Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.
Where it matters
Every search team's offline evaluation, and the reason interleaving experiments beat naive CTR comparisons.
Common mistakes
- Treating click-through rate as a relevance label and locking in the incumbent ranking
- Estimating position bias from the same log the ranker produced, with no randomisation to identify it
- Ignoring that a click is not satisfaction — a fast return to the results page is evidence against it
- Comparing CTR across layouts, where the examination probability changed too
Learn from primary sources
Use the linked roadmap context and practice prompt.
Practice and explain it back
Debias clicks by inverse propensity
Implement debiasedCtr(events, propensity) where events are {docId, position, clicked} and propensity maps position -> probability of examination. Return docId -> estimated relevance = (sum of clicks / propensity[position]) / (number of impressions). Round each value to 4 decimals.
Expected evidence: A doc clicked 1/10 times at position 1 (p=1.0) scores lower than one clicked 1/10 times at position 5 (p=0.2).
Open the interactive drill →Review prompts
- A team trains its ranker on click-through rate and the ranking gets more entrenched each cycle. Explain the loop and the standard correction.
Build evidence
Use a roadmap capstone to turn this concept into working evidence.