Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Importance of preference refinement <s> Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole. In addition to reviewing the evaluation strategies used by prior researchers, we present empirical results from the analysis of various accuracy metrics on one content domain where all the tested metrics collapsed roughly into three equivalence classes. Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated. <s> BIB001 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Importance of preference refinement <s> Two music recommender websites, Pandora (a content-based recommender) and Last.fm (a rating-based social recommender), were compared side-by-side in a within-subject user study involving 64 participants. The main objective was to investigate users’ initial adoption of recommender technology and their subjective perception of the respective systems. Results show that a simple interface design, the requirement of less initial effort, and the quality of recommended items (accuracy, novelty and enjoyability) are some of the key design features that such websites rely on to break the initial entrance barrier in becoming a popular website. <s> BIB002
Viappiani et al. (2007) conducted a user study with 22 subjects that compared the number of preferences elicited before and after the critiquing process. The average number of preferences stated was 2.1 on a total of 10 attributes, but this increased to 4.19 attributes when users came to the stage of selecting the final choice. The results suggest that an effective recommender mechanism should allow users to refine their preferences. Furthermore, as  indicated, the initial recommended results are not likely to be 100% accurate, so it is important to design a system that does not look like a dead-end. For rating-based systems, profile refinement has mainly been handled by requiring users to provide more ratings BIB001 ). For example, in Pandora (introduced previously), users are given the possibility to give a thumb up or down next to each of recommended songs BIB002 . For case-based systems and utility-based systems, this issue is addressed by initiating a preference critiquing support that enhances the refinement of features through a successive series of interactions . In commercial sites, Amazon employs a simple preference refinement method. It asks users to rate some specific items under the box that says "Improve Your Recommendations". This facility may convince users that their work will lead to more accurate recommendations and encourage them to put forth more effort. Therefore, we propose: