Recommendation systems are often evaluated with blackinteraction data blackobtained from users that have already been exposed to a deployed algorithm. This creates a feedback loop [1]}, [2]} (a.k.a. algorithmic confounding [3]} or closed loop feedback [4]}).
Using simulations on synthetic data, Chaney et al. [3]} revealed that this feedback loop in recommendation models blackencourages similar users to interact with the same set of items, blackthereby homogenising their behavior, relative to blackan open loop (randomised) platform blackwhere random items are exposed to the user.
In contrast to the simulation experiments, Jadidinejad et al. [4]} investigated the effect of closed loop feedback in both the training and offline evaluation of recommendation models based on blackan blackopen loop (randomised) dataset. They found that there is a strong correlation between the deployed model blackfrom which the blackusers' feedback were collected and a new model that is trained or evaluated based on the collected feedback dataset. For example, when popular items are favoured by the deployed algorithm, the observed interactions are biased towards popular items. In this situation, the performance of any algorithm that has a tendency to recommend popular items might be over-estimated [3]}, [8]}, [4]}.
blackIn this paper, we propose a theoretical framing of feedback loops in recommendation systems based on the Simpson's paradox, blackas well as formulate and validate hypotheses about the blackimplications of closed loop feedback in the offline evaluation of recommendation systems. In addition, we blackpropose a novel evaluation methodology, blackwhich takes into account the confounding role of the deployed model blackand show its benefits in comparison to both the blackstandard offline holdout evaluation [10]} and the recently introduced counterfactual evaluation [11]}, [12]} methods.
