
We present an in-depth analysis of the blackstandard offline evaluation of recommender systems, showing that it blackindeed suffers from blackthe Simpson's paradox.
Our in-depth experiments on blackfour well-known closed loop and open loop (randomised) datasets reveal that a blackvery small minority of items that are over-exposed by the deployed model play a confounding factor in blacka standard offline evaluation, i.e. significant trends in a blackgiven recommender system's performance that blackare observed in the majority of user-item interactions actually disappear or even reverse in the blackstandard offline evaluation when we merge this majority feedback with a minority confounder.

To address this problem, we propose a blacknew propensity-based stratified evaluation method in comparison to blackboth the blackstandard offline holdout evaluation [1]} and blackthe recently introduced counterfactual evaluation [2]}, [3]} methods. Our experiments on a broad range of recommendation models and evaluation metrics reveal that the proposed propensity-based stratified evaluation method better represents the actual utility of the examined recommendation models.

