The first key step in evaluating recommendation models based on feedback datasets is to understand how the feedback data was generated.
As for any causal analysis, we started from a hypothesis about the data generating process (Hypothesis REF ).
We investigated the influence of the deployed recommendation model on the closed loop feedback data generation blackprocess. blackNext, we studied the impact of this collected closed loop feedback data on the offline evaluation of a new recommendation model.
Our in-depth experiments based on both the closed loop and open loop (randomised) feedback datasets blackrevealed that the blackstandard holdout evaluation blackmethod suffers from the Simpson's paradox, i.e. a blackvery small minority of items, which are repeatedly exposed by the deployed model blackplays a confounding factor in the holdout evaluation where all feedback blackis aggregated together without accounting for the deployed blackmodel's confounding effect.
Consequently, our experiments blackhighlighted that when evaluating a recommendation model in an offline fashion, the test set should be chosen carefully in order to free blackit from any confounders, blackand specifically blackfrom the closed loop algorithmic effect. Our findings blackon blackfour well-known closed loop and open loop (randomised) feedback datasets provided a theoretical and experimental explanation of the blackconcerns regarding the offline evaluation of recommendation systems observed in the recent literature [1]}, [2]}, [3]}, [4]}, [5]}.
In addition, we proposed a novel propensity-based stratified evaluation blackmethod, which takes into account the role of the deployed model in the offline evaluation blackto more accurately estimate the performance of a given new recommendation model. The proposed model significantly blackoutperforms the classical holdout evaluation and the counterfactual evaluation methods blackfor deeper nDCG cut-offs, i.e. the proposed model better represents the actual relative order of the examined models blackusing an open loop blacksetup.
Our work aims blackto bring attention to the challenges of the offline evaluation based on closed loop feedback datasets and blackto show how the application of causal reasoning techniques, blackthrough our proposed propensity-based stratified evaluation method, blackallows to counter the confounding effect of the deployed model blackso as to obtain a more accurate evaluation of a recommender system.
