where \(R\)  is the actual performance of the predicted blackranking blackof items \(\hat{Z}\) . In this situation, the effectiveness of any model that captures the bias in the original dataset might be over-estimated [1]}, [2]}. For example, when popular items are favoured by the deployed algorithm, the observed interactions are biased towards popular items. blackAs a result, the performance of any algorithm that has a tendency to recommend popular items might be over-estimated [3]}.
An expensive solution is to create an open loop (randomised) test set from blackthe experiments, i.e. blackto ask blackthe users to provide feedback for randomly selected items. blackAnother alternative and more practical solution consists in using blackthe blackpropensity-based methods [4]} (e.g. counterfactual evaluation)blackblack, which we describe in the following section.
