Further experiments on the feedback sub-populations (blackthe Q1 and Q2 blackstrata in Table REF ) reveal that the offline evaluation of the examined models, blackwhich is only based on blackthe long-tail feedback (Q1 stratum) better correlates with the open loop evaluation.
blackSpecifically, for the Coat dataset, the offline evaluation of the examined models based on the Q2 stratum has a negative correlation with the open loop evaluation.
blackThis supports previous research blacksuch as Cremonesi et al. [1]}, blackwho found that the very few top popular items skew the rank-based evaluation of blackthe recommendation systems. They suggested to exclude the extremely popular items while evaluating the effectiveness of recommendation models.
