Typically, the offline evaluation of recommendation systems has two stages: (1) gathering a collection of user's feedback from a deployed system; and (2) the use of such feedback to empirically evaluate and compare different recommendation models.
The first stage can be subject to different types of confounders, either initiated by the item selection behaviour of users or through the actions of the deployed recommendation system [1]}, [2]}, [3]}.
For instance, blackaside from other interaction mechanisms such as search, users are more likely to interact with those items that they have been exposed to.
In this way, blackusing historical interactions in the offline evaluation of a new recommender model, where those interactions have been blackobtained from the deployed recommender system, blackforms blacka closed blackloop feedback [1]}, [5]}, i.e. the deployed recommender systemblackNaturally, a recommender system relies on a recommendation model or algorithm. Hence, in this paper, we interchangeably use system and model. has a direct blackeffect on the collected feedback, which is blackin turn used for the offline evaluation of other recommendation models.
blackblackHence, the new recommendation model is blackevaluated in terms of how much it tends to mimic the interactions collected by the deployed model, rather than blackhow well it meets blackthe true preferences of users. On the other hand, in an open loop (randomised) scenario, the deployed model is a random recommendation model, i.e. random items are exposed to the blackusers. Therefore, the feedback loop between the deployed model and the new recommendation model is broken and the deployed model has no effect on the collected feedback dataset blackand, consequently, it has also no effect on the offline evaluation of any new model based on the collected feedback dataset.
blackHowever, it blackis naturally not practical to expose users to random items blackdue to the potential for degraded user experience. Therefore, blackthe training and evaluation of recommendation systems based on closed loop feedback is a critical problem.
