We use blackfour datasets, which are widely used in the offline evaluation of recommendation system, namely blackthe MovieLens [1]}, blackNetflix [2]}, Yahoo! music rating [3]} blackand Coat shopping [4]} datasets.
MovieLens contains 1m ratings from 6k users and 4k itemsblack, Netflixhttps://www.kaggle.com/netflix-inc/netflix-prize-data/ contains 607k ratings from 10k users and 5k items, while blackthe Yahoo! dataset contains 300K ratings given by 15.4k users for 1k items.
blackFurthermore, the Coat shopping dataset simulates the shopping for a coat in an online store. It contains about 7k ratings given by 290 users for 300 items.
blackAll datasets were collected from an unknown deployed recommendation system in a closed loop scenario.
In addition, blackthe Yahoo! dataset has a unique feature: a subset of users (black5.4k) were asked to rate 10 randomly selected items (open loop scenario). blackTherefore, for a subset of users (5.4k), we have both blackthe closed loop and open loop (random) feedback.
blackSimilarly, for the Coat dataset, each user blackwas asked to provide ratings for 24 self-selected (closed loop) items and 16 randomly picked (open loop) items.
blackHowever, blackin a real-world practical evaluation setup, blackthe training of the models blackshould be based on the blackcollected closed loop feedback, blackwhile the actual evaluation blackcould be based on blackthe open loop (random) feedback blackblackbecause of the blackdifficulties of collecting open loop feedback in blackthe real-world, blackas discussed in Section 1.
blackWe use the blacknormalised Discounted Cumulative Gain (nDCG@k) metric for different cut-offs (\(k=\lbrace 5, 10, 20, 30, 100\rbrace \) )black, while blackwe use nDCG, without a rank cutoff, blackto denote the case where the metric is computed on the ranking of all items for each user.
blackThe user's rating value for blackall datasets is an integer \(r\in [0,5]\) . Our objective is to rank the most relevant items for each user. blackHence, we binarise all explicit rating values by keeping the highly recommended items (\(r\ge 4\) ).
blackIn the closed loop evaluation of the examined models, we use 80% of the closed loop dataset (MovieLens, Yahoo! and Coat) for training and a 20% random split for testing, blackas is typical in RecSys evaluation [5]}. On the other hand, for the open loop evaluation (Yahoo! and Coat), we evaluate the trained models on all the provided open loop (randomised) feedback.
