advanced practice · coding problem
Debias clicks by inverse propensity
Implement debiasedCtr(events, propensity) where events are {docId, position, clicked} and propensity maps position -> probability of examination. Return docId -> estimated relevance = (sum of clicks / propensity[position]) / (number of impressions). Round each value to 4 decimals.
The mechanism
A click means the user examined the result AND judged it worth clicking, so the observed rate confounds relevance with position — rank 1 gets clicked far more than rank 5 for identical content. Training on raw clicks therefore teaches the model to reproduce the ranking that generated them, a feedback loop that entrenches whatever was already on top. Inverse-propensity weighting breaks it by dividing each click by the estimated probability that the position was examined, which is why deliberate result randomisation is worth its cost.
Expected evidence
A doc clicked 1/10 times at position 1 (p=1.0) scores lower than one clicked 1/10 times at position 5 (p=0.2).
Try this drill in the practice workspace →
Hints when you need them
- Weight each CLICK by 1/propensity, then divide by impressions — not by the sum of weights.
- The whole point is that a click from a rarely-examined position is stronger evidence.
Watch for these mistakes
- Treating click-through rate as a relevance label and locking in the incumbent ranking
- Estimating position bias from the same log the ranker produced, with no randomisation to identify it
- Ignoring that a click is not satisfaction — a fast return to the results page is evidence against it
- Comparing CTR across layouts, where the examination probability changed too