Table REF  blackshows the Kendall's \(\tau \)  rank correlation coefficient between the examined evaluation methods and the ground truth, i.e. open loop (randomised) evaluation.
blackOn analysing the table, we observe that the
\(\tau \)  correlation between the holdout evaluation and the open loop evaluation is medium-to-strong (\(0.599\le \tau \le 0.729\) ) for the Yahoo! dataset, and weaker (blackand \(-0.40 \le \tau \le 0.327\) ) for the Coat dataset, with correlations blackon both datasets rising slightly for larger cut-offs.
blackIndeed, we note that the Coat dataset used a different data generation process compared to the MovieLensblack, Netflix and Yahoo! datasets [1]} (blacki.e. making use of crowdsourcing users rather than real users). In addition, the number of examined users and items are lower than blackthose in the Yahoo! dataset.
blackThis finding blackagrees with previous research [2]}, [3]} blackwho found blackthat the blackoffline evaluation based on standard rank-based metrics
does not blacknecessarily correlate with blackthe online evaluation (A/B testing or interleaving) results and researchers have questioned blackthe validity of offline evaluation [4]}, [5]}, [6]}.
