Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Accuracy <s> Collaborative filtering has proven to be valuable for recommending items in many different domains. In this paper, we explore the use of collaborative filtering to recommend research papers, using the citation web between papers to create the ratings matrix. Specifically, we tested the ability of collaborative filtering to recommend citations that would be suitable additional references for a target research paper. We investigated six algorithms for selecting citations, evaluating them through offline experiments against a database of over 186,000 research papers contained in ResearchIndex. We also performed an online experiment with over 120 users to gauge user opinion of the effectiveness of the algorithms and of the utility of such recommendations for common research tasks. We found large differences in the accuracy of the algorithms in the offline experiment, especially when balanced for coverage. In the online experiment, users felt they received quality recommendations, and were enthusiastic about the idea of receiving recommendations in this domain. <s> BIB001 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Accuracy <s> Recommender systems use people's opinions about items in an information domain to help people choose other items. These systems have succeeded in domains as diverse as movies, news articles, Web pages, and wines. The psychological literature on conformity suggests that in the course of helping people make choices, these systems probably affect users' opinions of the items. If opinions are influenced by recommendations, they might be less valuable for making recommendations for other users. Further, manipulators who seek to make the system generate artificially high or low recommendations might benefit if their efforts influence users to change the opinions they contribute to the recommender. We study two aspects of recommender system interfaces that may affect users' opinions: the rating scale and the display of predictions at the time users rate items. We find that users rate fairly consistently across rating scales. Users can be manipulated, though, tending to rate toward the prediction the system shows, whether the prediction is accurate or not. However, users can detect systems that manipulate predictions. We discuss how designers of recommender systems might react to these findings. <s> BIB002 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Accuracy <s> Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole. In addition to reviewing the evaluation strategies used by prior researchers, we present empirical results from the analysis of various accuracy metrics on one content domain where all the tested metrics collapsed roughly into three equivalence classes. Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated. <s> BIB003 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Accuracy <s> In this work we present topic diversification, a novel method designed to balance and diversify personalized recommendation lists in order to reflect the user's complete spectrum of interests. Though being detrimental to average accuracy, we show that our method improves user satisfaction with recommendation lists, in particular for lists generated using the common item-based collaborative filtering algorithm.Our work builds upon prior research on recommender systems, looking at properties of recommendation lists as entities in their own right rather than specifically focusing on the accuracy of individual recommendations. We introduce the intra-list similarity metric to assess the topical diversity of recommendation lists and the topic diversification approach for decreasing the intra-list similarity. We evaluate our method using book recommendation data, including offline analysis on 361, !, 349 ratings and an online study involving more than 2, !, 100 subjects. <s> BIB004 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Accuracy <s> Users' critiques to the current recommendation form a crucial feedback mechanism for refining their preference models and improving a system's accuracy in recommendations that may better interest the user. In this paper, we present a novel approach to assist users in making critiques according to their stated and potentially hidden preferences. This approach is derived from our previous work on critique generation and organization techniques. Based on a collection of real user data, we conducted an experiment to compare our approach with three existing critique generation systems. Results show that our preference-based organization interface achieves the highest level of prediction accuracy in suggesting users' intended critiques and recommendation accuracy in locating users' target choices. In addition, it can potentially most efficiently save real users' interaction effort in decision making. <s> BIB005 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Accuracy <s> Recommendation agents (RAs) are software agents that elicit the interests or preferences of individual consumers for products, either explicitly or implicitly, and make recommendations accordingly. RAs have the potential to support and improve the quality of the decisions consumers make when searching for and selecting products online. They can reduce the information overload facing consumers, as well as the complexity of online searches. Prior research on RAs has focused mostly on developing and evaluating different underlying algorithms that generate recommendations. This paper instead identifies other important aspects of RAs, namely RA use, RA characteristics, provider credi'r, and user-RA interaction, which influence users' decision-making processes and outcomes, as well as their evaluation of RAs. It goes beyond generalized models, such as TAM, and identifies the RA-specific features, such as RA input, process, and output design characteristics, that affect users' evaluations, including their assessments of the usefulness and ease-of-use of RA applications. ::: ::: Based on a review of existing literature on e-commerce RAs, this paper develops a conceptual model with 28 propositions derived from five theoretical perspectives. The propositions help answer the two research questions: (1) How do RA use, RA characteristics, and other factors influence consumer decision making processes and outcomes? (2) How do RA use, RA characteristics, and other factors influence users' evaluations of RAs? By identifying the critical gaps between what we know and what we need to know, this paper identifies potential areas of future research for scholars. It also provides advice to information systems practitioners concerning the effective design and development of RAs. <s> BIB006 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Accuracy <s> A critiquing-based recommender system acts like an artificial salesperson. It engages users in a conversational dialog where users can provide feedback in the form of critiques to the sample items that were shown to them. The feedback, in turn, enables the system to refine its understanding of the user's preferences and prediction of what the user truly wants. The system is then able to recommend products that may better stimulate the user's interest in the next interaction cycle. In this paper, we report our extensive investigation of comparing various approaches in devising critiquing opportunities designed in these recommender systems. More specifically, we have investigated two major design elements which are necessary for a critiquing-based recommender system: critiquing coverage--one vs. multiple items that are returned during each recommendation cycle to be critiqued; and critiquing aid--system-suggested critiques (i.e., a set of critique suggestions for users to select) vs. user-initiated critiquing facility (i.e., facilitating users to create critiques on their own). Through a series of three user trials, we have measured how real-users reacted to systems with varied setups of the two elements. In particular, it was found that giving users the choice of critiquing one of multiple items (as opposed to just one) has significantly positive impacts on increasing users' decision accuracy (particularly in the first recommendation cycle) and saving their objective effort (in the later critiquing cycles). As for critiquing aids, the hybrid design with both system-suggested critiques and user-initiated critiquing support exhibits the best performance in inspiring users' decision confidence and increasing their intention to return, in comparison with the uncombined exclusive approaches. Therefore, the results from our studies shed light on the design guidelines for determining the sweetspot balancing user initiative and system support in the development of an effective and user-centric critiquing-based recommender system. <s> BIB007 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Accuracy <s> This paper systematically evaluates the user experience of a recommender system. Using both behavioral data and subjective measures of user experience, we demonstrate that choice satisfaction and system effectiveness increase when a system provides personalized recommendations (compared to the same system that provides random recommendations). We furthermore demonstrate that despite privacy issues, this higher choice satisfaction and system effectiveness increases users’ intention to provide feedback about their preference. Due to an intention-behavior gap, this may however not necessarily influence the users’ actual feedback behavior. <s> BIB008 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Accuracy <s> Our previous research indicates that using personality quizzes is a viable and promising way to build user profiles to recommend entertainment products Based on these findings, our current research further investigates the feasibility of using personality quizzes to build user profiles not only for an active user but also his or her friends We first propose a general method that infers users' music preferences in terms of their personalities Our in-depth user studies show that while active users perceive the recommended items to be more accurate for their friends, they enjoy more using personality quiz based recommenders for finding items for themselves Additionally, we explore if domain knowledge has an influence on users' perception of the system We found that novice users, who are less knowledgeable about music, generally appreciated more personality-based recommenders Finally, we propose some design issues for recommender systems using personality quizzes. <s> BIB009 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Accuracy <s> As e-commerce has evolved into its second generation, where the available products are becoming more complex and their abundance is almost unlimited, the task of locating a desired choice has become too difficult for the average user. Therefore, more effort has been made in recent years to develop recommender systems that recommend products or services to users so as to assist in their decision-making process. In this article, we describe crucial experimental results about a novel recommender technology, called the preference-based organization (Pref-ORG), which generates critique suggestions in addition to recommendations according to users' preferences. The critique is a form of feedback (“I would like something cheaper than this one”) that users can provide to the currently displayed product, with which the system may better predict what the user truly wants. We compare the preference-based organization technique with related approaches, including the ones that also produce critique candidates, but without the consideration of user preferences. A simulation setup is first presented, that identified Pref-ORG's significantly higher algorithm accuracy in predicting critiques and choices that users should intend to make, followed by a real-user evaluation which practically verified its significant impact on saving users' decision effort. <s> BIB010 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Accuracy <s> One critical question suggested by Web 2.0 is as follows: When is it better to leverage the knowledge of other users vs. rely on the product characteristic-based metrics for online product recommenders? Three recent and notable changes of recommender systems have been as follows: (1) a shift from characteristic-based recommendation algorithms to social-based recommendation algorithms; (2) an increase in the number of dimensions on which algorithms are based; and (3) availability of products that cannot be examined for quality before purchase. The combination of these elements is affecting users' perceptions and attitudes regarding recommender systems and the products recommended by them, but the psychological effects of these trends remain unexplored. The current study empirically examines the effects of these elements, using a 2 (recommendation approach: content-based vs. collaborative-based, within)x2 (dimensions used to generate recommendations: 6 vs. 30, between)x2 (product type: experience products (fragrances) vs. search products (rugs), between) Web-based study (N=80). Participants were told that they would use two recommender systems distinguished by recommendation approach (in fact, the recommendations were identical). There were no substantive main effects, but all three variables exhibited two-way interactions, indicating that design strategies must be grounded in a multi-dimensional understanding of these variables. The implications of this research for the psychology and design of recommender systems are presented. <s> BIB011 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Accuracy <s> Research on recommender systems typically focuses on the accuracy of prediction algorithms. Because accuracy only partially constitutes the user experience of a recommender system, this paper proposes a framework that takes a user-centric approach to recommender system evaluation. The framework links objective system aspects to objective user behavior through a series of perceptual and evaluative constructs (called subjective system aspects and experience, respectively). Furthermore, it incorporates the influence of personal and situational characteristics on the user experience. This paper reviews how current literature maps to the framework and identifies several gaps in existing work. Consequently, the framework is validated with four field trials and two controlled experiments and analyzed using Structural Equation Modeling. The results of these studies show that subjective system aspects and experience variables are invaluable in explaining why and how the user experience of recommender systems comes about. In all studies we observe that perceptions of recommendation quality and/or variety are important mediators in predicting the effects of objective system aspects on the three components of user experience: process (e.g. perceived effort, difficulty), system (e.g. perceived system effectiveness) and outcome (e.g. choice satisfaction). Furthermore, we find that these subjective aspects have strong and sometimes interesting behavioral correlates (e.g. reduced browsing indicates higher system effectiveness). They also show several tradeoffs between system aspects and personal and situational characteristics (e.g. the amount of preference feedback users provide is a tradeoff between perceived system usefulness and privacy concerns). These results, as well as the validated framework itself, provide a platform for future research on the user-centric evaluation of recommender systems. <s> BIB012
In the literature, accuracy has been used as a dominating criterion for evaluating recommender systems (McNee et al. 2006a,b; BIB003 . Over the past decade, a wide variety of algorithms have been developed with the goal of increasing objective accuracy BIB003 ). For example, Mean Absolute Error (MAE), defined as the difference between the predicted ratings of an algorithm and actual user ratings, is the commonly adopted measure to evaluate the objective accuracy of rating-based systems. For content-or feature-based recommenders, established a procedure, called the switching task, to measure the objective accuracy. Specifically, it measures whether a user changes his/her initially preferred item, which was identified by using a recommender agent, to another one when instructed to view all available items in a later round. If some users change their mind (switching), the accuracy of the recommender will be discounted. More recently, researchers have been paying attention to users' perceived accuracy. This measure is defined as the degree to which users feel proposed recommendations match their interests and preferences. Compared to objective algorithm accuracy, perceived accuracy has a more direct influence on users' trust formation with the system and behavioral intentions, such as their intention to purchase items recommended to them and intention to return to the system BIB007 ). Some studies indicate that user perception is directly correlated with objective differences in recommendation quality BIB004 BIB005 . BIB008 investigated the effects of two recommendation algorithms on users' perceived quality. Their results show that users given personalized recommendations scored higher in terms of perceived accuracy, in contrast to the other group of users given randomly selected items. The high correlation suggests that users are aware of the quality distinctions between random and personalized recommendations. These results have also been validated in a series of their follow-up studies BIB012 . However, some researchers have shown that objective accuracy may not have a direct correlation with perceived accuracy. BIB001 compared six recommendation algorithms through both offline and online experiments. In the offline experiment, they found that the "item-to-item" algorithm that achieved the best prediction performance gave the fewest recommendations that users believed to be relevant and helpful. On the other hand, perceived accuracy can be enhanced by other aspects, such as the level of domain knowledge of individuals BIB009 BIB012 , contexts and interface design BIB006 BIB002 BIB010 , and interface labels BIB011 ). The study of BIB002 shows that the display of predicted ratings could manipulate users' opinions on the recommended items, with a bias towards the presented rating values. In the experiment conducted by BIB011 , users were manipulated to think that they were evaluating two recommender system approaches by interface labels, when in reality they received the same recommendations. Their empirical results show that for search products (defined as those whose qualities can be determined before their purchase, e.g., rugs), participants tended to like the recommendations from collaborative-based systems, while for experience products (defined as those whose quality cannot be determined until after usage and purchase e.g., perfume), participants tended to feel better about the recommendations from content-based system. Considering the existence of the gap between objective accuracy and perceived accuracy, we propose: Guideline 14 Consider enhancing users' perceived accuracy with more attractive layout design and effective labels, and explaining how the systems compute the recommendations. Doing so can increase users' perception of the system's effectiveness, their overall satisfaction of the system, their readiness to accept the recommended items, and their trust in the system.