The holdout evaluation method is the blackstandard and most common evaluation method in the field of recommendation systems because of its simplicity and flexibility [1]}, [2]}. However, if the dataset blackis biased towards a group of items, the holdout test set will be a biased sample of the user-item interactions [3]}. Therefore, the holdout evaluation does not necessarily represent the actual effectiveness of the examined models, i.e. the expected outcome of the holdout evaluation does not blacknecessarily conform to the open loop evaluation [4]}:
\(\mathbb {E}_{O}\left[\hat{R}_{\mathrm {holdout}}(\hat{Z})\right] \ne R(\hat{Z})\) 
