To better understand the subtleties of blackthe tackled problem, consider a deployed recommendation model that promotes a specific group of items (e.g. popular itemsIt is well-known in the literature that recommendation algorithms and public datasets favor popular items [1]}, [2]}, [3]}.) – there are a few head items that are widely exposed to the users blackthat obtain blacka large number of interactions, in contrast to a long tail of items with blackonly a few interactions.
When we evaluate a new recommendation model based blackon feedback that blackwas collected from the blackaforementioned deployed model without taking into account black(or adjusting) how blackfrequently different items blackwere exposed, the evaluation process is confounded by the deployed model's blackcharacteristics, i.e. the performance of any model that has a tendency to blackexhibit similar properties to the already deployed model blackwill likely be over-estimated.
blackIn this situation, we might choose to deploy a new model that matches the properties of the already deployed model, while it might be less effective than another model blackfrom the blackactual users' perspective.
blackIn this paper, we thoroughly investigate the consequences of this problem on the conclusions drawn from the blackstandard offline evaluation of recommender systems and propose a new methodology to address the issue. blackIn particular, blackthe blackcontributions of this paper blackare blacktwo-fold:
