Although researchers in IR have proposed novel approaches for both learning [1]} and evaluating [2]}, [3]} from user's clickthrough data, many researchers and practitioners blackstill focus on evaluating recommendation systems in terms of accuracy metrics on datasets that were collected from an already deployed system [4]}, [5]}, [6]}, [7]}.
On the other hand, blackseveral recent studies [8]}, [9]}, [10]} blackhave attempted to estimate the actual utility of a recommendation model with propensity-weighting techniques commonly used in the off-policy evaluation approaches in Reinforcement Learning with the aim of achieving an unbiased performance estimator of biased observations.
blackPropensity in this context is the probability that the deployed model blackhas exposed item \(i\)  to user \(u\)  by the time of feedback collection.
The intuition blackin counterfactual evaluation is that each feedback is blackindividually weighted based on its propensity.
The key to a fair evaluation in blackthe counterfactual approaches lies in accurately logging propensities in blackan open loop (randomised) dataset where random items are exposed to the user. blackHowever, most current public recommendation datasets do not introduce a randomisation element [9]}, [10]}, blackmaking it practically difficult to effectively deploy the counterfactual approaches for evaluating recommender systems.
blackIn the absence of open loop (randomised) feedback, the estimated propensities are noisy and skewed. As a result, weighting each feedback individually based on its propensity score blackcan lead to blacka high variance blackin the estimation of blackthe model's effectiveness, as blackreported in blackseveral previous blackstudies [13]}, [8]}, [9]}, [10]}.
blackHence, compared to previous blackworks on propensity weighting [9]}, [10]}, our model does not weight each feedback individually based on its propensity. We blackinstead propose a novel evaluation method based on stratification, blackwhich blackblackconsiders each feedback in the corresponding stratum based on the estimated propensities.
As a result, blackas our experiments show, blackour proposed evaluation method is more robust blackin blackblackcoping with these noisy blackestimated propensities.
