Users' feedback in recommendation systems is reminiscent of clickthrough data in information retrieval (IR). Previous research [1]} has shown that there is a strong dependency between the documentsThese correspond to items in recommendation systems. presented to the user, and those for which the system receives feedback, i.e. higher-ranked documents obtain more clicks (blacka phenomenon denoted in the literature as position/presentation bias).
For example, consider a movie recommendation system that aims to learn the users' preferences by observing their watching history. The system can infer which blackmovie(s) they prefer over other programs at a particular time, but these are relative preferences in relation to the exposed movies, blackbut blackdo not constitute preferences on an absolute scale.
Previous research blackin information retrieval (IR) [2]}, [1]} has revealed that such feedback does not reliably reflect retrieval quality while blacknonetheless still providing relative blackuser preferences in paired experiments.
Classical blackoffline evaluation blacksetups in IR use pooling blackto gather a comprehensive set of possibly relevant documents for assessment, thereby preventing bias in the relevance assessments towards a single system (known as incompleteness).
However, pooling is not a suitable approach for addressing incompleteness in the evaluation of recommendation systems, due to the contextual and personalised nature of blackusers' recommendation needs.
