Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Novelty <s> Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole. In addition to reviewing the evaluation strategies used by prior researchers, we present empirical results from the analysis of various accuracy metrics on one content domain where all the tested metrics collapsed roughly into three equivalence classes. Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated. <s> BIB001 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Novelty <s> Recommender systems have shown great potential to help users find interesting and relevant items from within a large information space. Most research up to this point has focused on improving the accuracy of recommender systems. We believe that not only has this narrow focus been misguided, but has even been detrimental to the field. The recommendations that are most accurate according to the standard metrics are sometimes not the recommendations that are most useful to users. In this paper, we propose informal arguments that the recommender community should move beyond the conventional accuracy metrics and their associated experimental methodologies. We propose new user-centric directions for evaluating recommender systems. <s> BIB002 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Novelty <s> Two music recommender websites, Pandora (a content-based recommender) and Last.fm (a rating-based social recommender), were compared side-by-side in a within-subject user study involving 64 participants. The main objective was to investigate users’ initial adoption of recommender technology and their subjective perception of the respective systems. Results show that a simple interface design, the requirement of less initial effort, and the quality of recommended items (accuracy, novelty and enjoyability) are some of the key design features that such websites rely on to break the initial entrance barrier in becoming a popular website. <s> BIB003
As indicated above, novelty is another dimension that the designers of a RS should consider. The core concept of this dimension is the unexpectedness of recommendations for users. BIB002 also called this criterion "serendipity", which emphasizes the experience of receiving unexpected and fortuitous item recommendations. BIB001 argued that novelty is different from serendipity, because novelty only covers the concept "new", while serendipity means not only "new" but also "surprising". In this paper, we unify the two understandings and measure "novelty" as the extent to which users receive new and interesting recommendations. The effect of novelty has experimentally been demonstrated in music recommenders by BIB003 . They discovered that users found the songs of Pandora (a popular music recommender site that offers more novel recommendations) to be significantly more enjoyable and better than songs suggested by their friends. This user study also showed that users will more likely use recommender systems that make novel music recommendations, as such a system is perceived to be more useful. Thus, our suggestion in this regard is: Guideline 16 Consider achieving a proper balance between familiarity and novelty among recommendations, since novelty can be correlated to the perceived usefulness of the system, especially in entertainment product domains.