Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Sufficiency of information <s> Valid measurement scales for predicting user acceptance of computers are in short supply. Most subjective measures used in practice are unvalidated, and their relationship to system usage is unknown. The present research develops and validates new scales for two specific variables, perceived usefulness and perceived ease of use, which are hypothesized to be fundamental determinants of user acceptance. Definitions of these two variables were used to develop scale items that were pretested for content validity and then tested for reliability and construct validity in two studies involving a total of 152 users and four application programs. The measures were refined and streamlined, resulting in two six-item scales with reliabilities of .98 for usefulness and .94 for ease of use. The scales exhibited hgih convergent, discriminant, and factorial validity. Perceived usefulness was significnatly correlated with both self-reported current usage r = .63, Study 1) and self-predicted future usage r = .85, Study 2). Perceived ease of use was also significantly correlated with current usage r = .45, Study 1) and future usage r = .59, Study 2). In both studies, usefulness had a signficnatly greater correaltion with usage behavior than did ease of use. Regression analyses suggest that perceived ease of use may actually be a causal antecdent to perceived usefulness, as opposed to a parallel, direct determinant of system usage. Implications are drawn for future research on user acceptance. <s> BIB001 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Sufficiency of information <s> Automated collaborative filtering (ACF) systems predict a person's affinity for items or information by connecting that person's recorded interests with the recorded interests of a community of people and sharing ratings between like-minded persons. However, current recommender systems are black boxes, providing no transparency into the working of the recommendation. Explanations provide that transparency, exposing the reasoning and data behind a recommendation. In this paper, we address explanation interfaces for ACF systems - how they should be implemented and why they should be implemented. To explore how, we present a model for explanations based on the user's conceptual model of the recommendation process. We then present experimental results demonstrating what components of an explanation are the most compelling. To address why, we present experimental evidence that shows that providing explanations can improve the acceptance of ACF systems. We also describe some initial explorations into measuring how explanations can improve the filtering performance of users. <s> BIB002 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Sufficiency of information <s> Recommender Systems act as a personalized decision guides, aiding users in decisions on matters related to personal taste. Most previous research on Recommender Systems has focused on the statistical accuracy of the algorithms driving the systems, with little emphasis on interface issues and the user's perspective. The goal of this research was to examine the role of transprency (user understanding of why a particular recommendation was made) in Recommender Systems. To explore this issue, we conducted a user study of five music Recommender Systems. Preliminary results indicate that users like and feel more confident about recommendations that they perceive as transparent. <s> BIB003 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Sufficiency of information <s> Under-contribution is a problem for many online communities. Social psychology theories of social loafing and goal-setting can provide mid-level design principles to address this problem. We tested the design principles in two field experiments. In one, members of an online movie recommender community were reminded of the uniqueness of their contributions and the benefits that follow from them. In the second, they were given a range of individual or group goals for contribution. As predicted by theory, individuals contributed when they were reminded of their uniqueness and when they were given specific and challenging goals, but other predictions were not borne out. The paper ends with suggestions and challenges for mining social science theories as well as implications for design. <s> BIB004 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Sufficiency of information <s> Marketers have been challenged by proponents of individual (one-to-one) marketing to shift from focusing on market segments to making individually customized offers. Building on current knowledge regarding the construction of customers’ preferences, the author examines the basic assumptions underlying individual marketing and presents a process model of customers’ responses to customized offers. The model addresses (1) preference development, (2) evaluation of customized offers, (3) likelihood of purchasing the customized offers, and (4) maintenance of relationships with one-to-one marketers. The analysis leads to specific propositions regarding determinants of customers’ responses to customized offers. The author discusses future research directions and managerial implications. <s> BIB005 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Sufficiency of information <s> Based on our recent work on the development of a trust model for recommender agents and a qualitative survey, we explore the potential of building users' trust with explanation interfaces. We present the major results from the survey, which provided a roadmap identifying the most promising areas for investigating design issues for trust-inducing interfaces. We then describe a set of general principles derived from an in-depth examination of various design dimensions for constructing explanation interfaces, which most contribute to trust formation. We present results of a significant-scale user study, which indicate that the organization-based explanation is highly effective in building users' trust in the recommendation interface, with the benefit of increasing users' intention to return to the agent and save cognitive effort. <s> BIB006 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Sufficiency of information <s> Recommendation agents (RAs) are software agents that elicit the interests or preferences of individual consumers for products, either explicitly or implicitly, and make recommendations accordingly. RAs have the potential to support and improve the quality of the decisions consumers make when searching for and selecting products online. They can reduce the information overload facing consumers, as well as the complexity of online searches. Prior research on RAs has focused mostly on developing and evaluating different underlying algorithms that generate recommendations. This paper instead identifies other important aspects of RAs, namely RA use, RA characteristics, provider credi'r, and user-RA interaction, which influence users' decision-making processes and outcomes, as well as their evaluation of RAs. It goes beyond generalized models, such as TAM, and identifies the RA-specific features, such as RA input, process, and output design characteristics, that affect users' evaluations, including their assessments of the usefulness and ease-of-use of RA applications. ::: ::: Based on a review of existing literature on e-commerce RAs, this paper develops a conceptual model with 28 propositions derived from five theoretical perspectives. The propositions help answer the two research questions: (1) How do RA use, RA characteristics, and other factors influence consumer decision making processes and outcomes? (2) How do RA use, RA characteristics, and other factors influence users' evaluations of RAs? By identifying the critical gaps between what we know and what we need to know, this paper identifies potential areas of future research for scholars. It also provides advice to information systems practitioners concerning the effective design and development of RAs. <s> BIB007 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Sufficiency of information <s> We conducted a user study evaluating two preference elicitation approaches based on ratings and personality quizzes respectively. Three criteria were used in this comparative study: perceived accuracy, user effort and user loyalty. Results from our study show that the perceived accuracy in two systems is not significantly different. However, users expended significantly less effort, both perceived cognitive effort and actual task time, to complete the preference profile establishing process in the personality quiz-based system than in the rating-based system. Additionally, users expressed stronger intention to reuse the personality quiz-based system and introduce it to their friends. After using these two systems, 53% of users preferred the personality quiz-based system vs. 13% of users preferred the rating-based system, since most users thought the former is easier to use. <s> BIB008 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Sufficiency of information <s> Critiquing-based recommenders do not require users to state all of their preferences upfront or rate a set of previously experienced products. Compared to other types of recommenders, they require relatively little user effort, especially initially, despite potential accuracy problems. On the other hand, they rely on a set of critiques to elicit users feedback in order to improve accuracy. Thus the better the critiques are, the more accurately and efficiently the system becomes in generating its recommendations. This method has been successfully applied to high-involvement products. However, it was never tested on public taste products such as music, films, perfumes, fashion goods or wine. Indeed our initial trial adapting traditional critiquing methods to this new domain led to unsatisfactory results. This has motivated us to develop a novel approach named "editorial picked critiques" (EPC) that accounts for users' needs for popularity information, editorial suggestions, as well as their needs for personalization and diversity. Through an empirical study, we demonstrate that EPC presents a viable recommender approach and is superior on several dimensions to critiques generated by data mining methods. <s> BIB009 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Sufficiency of information <s> The evaluation of interactive adaptive systems has long been acknowledged to be a complicated and demanding endeavour. Some promising approaches in the recent past have attempted tackling the problem of evaluating adaptivity by "decomposing" and evaluating it in a "piece-wise" manner. Separating the evaluation of different aspects can help to identify problems in the adaptation process. This paper presents a framework that can be used to guide the "layered" evaluation of adaptive systems, and a set of formative methods that have been tailored or specially developed for the evaluation of adaptivity. The proposed framework unifies previous approaches in the literature and has already been used, in various guises, in recent research work. The presented methods are related to the layers in the framework and the stages in the development lifecycle of interactive systems. The paper also discusses practical issues surrounding the employment of the above, and provides a brief overview of complementary and alternative approaches in the literature. <s> BIB010 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Sufficiency of information <s> Research on recommender systems typically focuses on the accuracy of prediction algorithms. Because accuracy only partially constitutes the user experience of a recommender system, this paper proposes a framework that takes a user-centric approach to recommender system evaluation. The framework links objective system aspects to objective user behavior through a series of perceptual and evaluative constructs (called subjective system aspects and experience, respectively). Furthermore, it incorporates the influence of personal and situational characteristics on the user experience. This paper reviews how current literature maps to the framework and identifies several gaps in existing work. Consequently, the framework is validated with four field trials and two controlled experiments and analyzed using Structural Equation Modeling. The results of these studies show that subjective system aspects and experience variables are invaluable in explaining why and how the user experience of recommender systems comes about. In all studies we observe that perceptions of recommendation quality and/or variety are important mediators in predicting the effects of objective system aspects on the three components of user experience: process (e.g. perceived effort, difficulty), system (e.g. perceived system effectiveness) and outcome (e.g. choice satisfaction). Furthermore, we find that these subjective aspects have strong and sometimes interesting behavioral correlates (e.g. reduced browsing indicates higher system effectiveness). They also show several tradeoffs between system aspects and personal and situational characteristics (e.g. the amount of preference feedback users provide is a tradeoff between perceived system usefulness and privacy concerns). These results, as well as the validated framework itself, provide a platform for future research on the user-centric evaluation of recommender systems. <s> BIB011 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Sufficiency of information <s> This research was motivated by our interest in understanding the criteria for measuring the success of a recommender system from users' point view. Even though existing work has suggested a wide range of criteria, the consistency and validity of the combined criteria have not been tested. In this paper, we describe a unifying evaluation framework, called ResQue (Recommender systems' Quality of user experience), which aimed at measuring the qualities of the recommended items, the system's usability, usefulness, interface and interaction qualities, users' satisfaction with the systems, and the influence of these qualities on users' behavioral intentions, including their intention to purchase the products recommended to them and return to the system. We also show the results of applying psychometric methods to validate the combined criteria using data collected from a large user survey. The outcomes of the validation are able to 1) support the consistency, validity and reliability of the selected criteria; and 2) explain the quality of user experience and the key determinants motivating users to adopt the recommender technology. The final model consists of thirty two questions and fifteen constructs, defining the essential qualities of an effective and satisfying recommender system, as well as providing practitioners and scholars with a cost-effective way to evaluate the success of a recommender system and identify important areas in which to invest development resources. <s> BIB012
Finally, it is crucial to provide sufficient information about the recommended items in order to facilitate users' decision-making processes. BIB003 suggested that the presence of longer descriptions of individual items correlates positively with both the perceived usefulness and ease of use of the recommender system. This suggests that users like to have more information about the recommended item, such as book description, author information, genre information and other users' reviews. Their research reinforced this finding by examining the difference between the two versions of RatingZone (one of their sample prototypes). The first version of RatingZone's Quick Picks showed only the book title and author name in the list of recommendations. The second version of RatingZone included simply the title and author, as well as providing a link to item-specific information at Amazon. The perceived usefulness of the second version was almost three times higher than the first. BIB003 also provided several types of information users found useful: basic item information (i.e., song, album, artist name, genre information, the time when album was released, the album cover); expert and community ratings (reviews and ratings provided by other users); item sample (e.g., audio clips of songs, sample chapters of books). If the designer does not have access to detailed item information, offering a community forum with other users' comments can be a relatively easy way to increase the system's efficacy . They also suggested organizing the information in an obvious way by using an effective navigation structure; when detailed information was offered, users had trouble actually finding it, since locating item information demanded several mouse clicks and the site had poor navigation design. Given these results, we suggest: Guideline 20 Consider providing sufficient information related to the recommended items and controlling the information's quality and navigation structure. 6 Related work BIB011 provided a framework to explain how objective system aspects (such as its input, process and output) influence users' perception of these aspects, and how this perception eventually influences users' choice satisfaction and their intent to provide feedback (see Fig. 4 ). A set of six structurally related constructs were proposed in their framework: objective recommender system aspects, subjective evaluations, subjective experiences, objective behaviors, situational, and personal characteristics. The objective system aspects refer to the input, output and process aspects of a system, including the visual and interaction design, the recommender algorithm, the feedback mechanisms, the recommendation presentation, and additional system features (e.g., social networking). The subjective system aspects are users' evaluations of the objective system aspects. The situational (different tasks) and personal characteristics (general trust, control and social factors) are external factors. The subjective experience also represents an evaluation of the system by the user, but User-centric evaluation framework for recommender systems BIB011 it may primarily depend on personal characteristics. The interaction is objectively measurable, and determines users' behavior in the system. For example, a system that is evaluated positively will be used more extensively. The model postulates that objective system aspects influence users' subjective evaluation, and eventually affect users' experience and objective interactions. Furthermore, situational and personal characteristics could mediate this process. A set of six user studies, four field studies and two controlled experiments, were conducted using this framework. The main findings from these studies show that personalized recommendations (versus randomly selected items) have higher perceived recommendation quality which causes higher perceived system effectiveness and choice satisfaction; despite privacy issues, higher choice satisfaction and system effectiveness can increase users' intention to provide feedback about their preferences; higher perceived recommendation quality leads to less perceived cognitive effort of using the system, and less cognitive effort causes higher perceived effectiveness and fun, which in turn increases users' choice satisfaction; and a higher perceived variety of recommendations can increase the perceived recommendation quality. The main limitation is that the outcomes were limited to the specific recommender systems (multimedia recommenders) used in all of the six experiments and the specific algorithms used to implement these systems. While the design and procedure can serve as illustrative examples of future work in this area, if some of the system's aspects were to change, users might also alter their perceptions and behavioral intentions. In addition, an important aspect of system, the recommender system's ability to explain its result, was not considered and behaviors are limited to users' intention to provide feedback (preference revision). Controlled experiments, such as those described in BIB011 , are often used to investigate and understand one or a few aspects of the system's interface and interaction design. This method is called layered evaluation in BIB010 . It proposed systematically guiding the evaluations of adaptive systems by breaking down a system into its "constituents" and evaluating each of these constituents separately. Although the layered evaluation method has not been applied to RS, it can be used as a powerful technique in identifying areas of RS that require more focused future work. In contrast to layered evaluation and controlled experiments, summative evaluations aim at examining the overall impact of the system on users' behavior and explaining users' motivations for using the system. Some researchers have recently started developing evaluation frameworks to measure the overall quality and benefits of a recommender system to its users. Based on an extensive survey of existing usability experience research on recommender systems, proposed a research model to understand the motivations of RS use including intentions to use the system and purchase the recommended items. This model is called ResQue (Recommender system's Quality of user experience), and was developed based on TAM theories BIB001 ). ResQue's constructs (composed of 43 question items) are categorized into four main dimensions: perceived system qualities (such as the quality of recommended items, the interaction characteristics to handle user preferences, the interface qualities to present and explain results, and to provide information sufficiency), users' beliefs as a result of perceived qualities (the beliefs include perceived ease of use, perceived usefulness, control and transparency), users' subjective attitudes (including overall satisfaction, confidence, and trust), and their behavioral intentions (including users' agreement to use the system, purchase intention, return intention and intention to introduce this system to friends). Recently, BIB012 reported a large-scale Web-based survey which successfully validated ResQue in terms of its structural consistency, validity and reliability, and the model's fitness with respect to the original hypotheses (see Fig. 5 ). One of the major extensions to TAM is that two types of users' behavioral intentions, i.e., purchase intention and use intention, have been identified to explain RS' role in e-commerce, and entertainment websites. That is, while overall satisfaction of the system (also in terms of ease of use and perceived usefulness) is important for use intention, trust of the system and choice confidence are fundamental for purchase intentions. ResQue is the first validated framework providing a well-balanced set of usercentric criteria for measuring the success of a recommender system and explaining user adoption of RS by providing an account of how the perceived qualities of a RA influence user beliefs and attitudes, and inspire users' behavioral intentions. The framework merged divergent criteria in past user experience research of RS and, via validation, produced a model defining the essential qualities of an effective and satisfying recommender system. ResQue can be presented as a long or a short version of an assessment questionnaire, 43 and 14 questions respectively, that can help designers and researchers forecast the adoption of recommender systems. The model can be applied to assess different types of recommenders, including rating-based, utility-based, and knowledge-based systems, regardless of the backend engines used. In comparison with BIB011 , the ResQue framework and its outcome were validated independently of the underlying recommender systems or the algorithms. On the other hand, the model proposed in BIB011 accounts for personal characteristics the users and contextual characteristics of RS use. BIB007 surveyed previous theoretical and empirical studies of Recommendation Agents (RAs) in both online and offline shopping environments, mainly based on existing literature in management of information systems (MIS). It focused on the RA's function to assist consumers in deciding which products to purchase, as well as how the use of RA, RA properties, and other factors influence consumers' evaluations of RAs. The summary of results were combined into a conceptual model of 28 hypotheses using theories of human information processing, interpersonal similarity, trust formation, technology acceptance model (TAM), and satisfaction. The majority of the 28 hypotheses were supported by corresponding empirical work. For example, RA use improved consumers' decision quality; the explicit preference elicitation method leads to better decision quality and higher decision effort than does the implicit preference elicitation method; recommending more alternatives increased information searched, decreased the quality of the consideration set, led to poor product choices, and reduced consumers' selectivity; ease of generating new or additional recommendations improved estimations of the ease of use of an RA; increased user control over interaction with the RA resulted in increased users' trust in and satisfaction; and explanations of an RA's inner logic strengthened users' trusting beliefs in the RA's competence and benevolence. The main limitation, as outlined by the authors, was the lack of results to support user behavioral intentions such as the intention to use RA in the future and the intention to purchase/accept the items recommended to them. The reviewed work was also limited to existing RA methods that were employed in existing online environments. Innovative techniques experimented in research laboratories were not included. For example, recent work on trust-inspiring interfaces, critiquing based RAs which offer compensatory decision aid, and diversity-compatible recommenders were not examined, although these results are covered in this article. In addition, while the summary from BIB007 provided recommendations relating features of RA or RA types to users' decision outcome and their evaluations of RAs, the 28 hypotheses did not provide suggestive design guidelines Hu and Pu (2009a,b) Personality-based recommenders; tested for movie and music Guideline 6: consider using entertaining personality quizzes as a preference elicitation method Comparison of elicitation methods BIB008 To all types Guideline 7: conduct comparative user studies to identify the optimal elicitation method balancing between accuracy and effort User contribution BIB004 To all types Guidelines 8 and 9: make users aware of their unique contributions, and set specific goals for users to achieve  Tested in content-based recommenders for entertainment products Guideline 16: achieve a proper balance between familiarity and novelty BIB002 , Swearingen (2001, 2002) , BIB005 , Tintarev and Masthoff (2007a,b) , BIB006 , BIB009 To all types and tested for both low-risk (e.g., music, movie) and high-risk products (e.g., laptops, digital cameras) Guideline 19: explain why the system recommends the suggested items to increase transparency and explain tradeoffs to increase persuasion Sufficiency of information BIB003 To all types and tested for books and songs Guideline 20: provide sufficient information related to the recommendation items on how to design and implement an effective RA. Therefore, the survey provided in this paper can be seen as a complimentary work.