Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Explanation of recommendations <s> Automated collaborative filtering (ACF) systems predict a person's affinity for items or information by connecting that person's recorded interests with the recorded interests of a community of people and sharing ratings between like-minded persons. However, current recommender systems are black boxes, providing no transparency into the working of the recommendation. Explanations provide that transparency, exposing the reasoning and data behind a recommendation. In this paper, we address explanation interfaces for ACF systems - how they should be implemented and why they should be implemented. To explore how, we present a model for explanations based on the user's conceptual model of the recommendation process. We then present experimental results demonstrating what components of an explanation are the most compelling. To address why, we present experimental evidence that shows that providing explanations can improve the acceptance of ACF systems. We also describe some initial explorations into measuring how explanations can improve the filtering performance of users. <s> BIB001 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Explanation of recommendations <s> Recommender Systems act as a personalized decision guides, aiding users in decisions on matters related to personal taste. Most previous research on Recommender Systems has focused on the statistical accuracy of the algorithms driving the systems, with little emphasis on interface issues and the user's perspective. The goal of this research was to examine the role of transprency (user understanding of why a particular recommendation was made) in Recommender Systems. To explore this issue, we conducted a user study of five music Recommender Systems. Preliminary results indicate that users like and feel more confident about recommendations that they perceive as transparent. <s> BIB002 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Explanation of recommendations <s> Marketers have been challenged by proponents of individual (one-to-one) marketing to shift from focusing on market segments to making individually customized offers. Building on current knowledge regarding the construction of customers’ preferences, the author examines the basic assumptions underlying individual marketing and presents a process model of customers’ responses to customized offers. The model addresses (1) preference development, (2) evaluation of customized offers, (3) likelihood of purchasing the customized offers, and (4) maintenance of relationships with one-to-one marketers. The analysis leads to specific propositions regarding determinants of customers’ responses to customized offers. The author discusses future research directions and managerial implications. <s> BIB003 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Explanation of recommendations <s> Based on our recent work on the development of a trust model for recommender agents and a qualitative survey, we explore the potential of building users' trust with explanation interfaces. We present the major results from the survey, which provided a roadmap identifying the most promising areas for investigating design issues for trust-inducing interfaces. We then describe a set of general principles derived from an in-depth examination of various design dimensions for constructing explanation interfaces, which most contribute to trust formation. We present results of a significant-scale user study, which indicate that the organization-based explanation is highly effective in building users' trust in the recommendation interface, with the benefit of increasing users' intention to return to the agent and save cognitive effort. <s> BIB004 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Explanation of recommendations <s> This paper characterizes general properties of useful, or Effective, explanations of recommendations. It describes a methodology based on focus groups, in which we elicit what helps moviegoers decide whether or not they would like a movie. Our results highlight the importance of personalizing explanations to the individual user, as well as considering the source of recommendations, user mood, the effects of group viewing, and the effect of explanations on user expectations. <s> BIB005 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Explanation of recommendations <s> While recommender systems tell users what items they might like, explanations of recommendations reveal why they might like them. Explanations provide many benefits, from improving user satisfaction to helping users make better decisions. This paper introduces tagsplanations, which are explanations based on community tags. Tagsplanations have two key components: tag relevance, the degree to which a tag describes an item, and tag preference, the user's sentiment toward a tag. We develop novel algorithms for estimating tag relevance and tag preference, and we conduct a user study exploring the roles of tag relevance and tag preference in promoting effective tagsplanations. We also examine which types of tags are most useful for tagsplanations. <s> BIB006 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Explanation of recommendations <s> As e-commerce has evolved into its second generation, where the available products are becoming more complex and their abundance is almost unlimited, the task of locating a desired choice has become too difficult for the average user. Therefore, more effort has been made in recent years to develop recommender systems that recommend products or services to users so as to assist in their decision-making process. In this article, we describe crucial experimental results about a novel recommender technology, called the preference-based organization (Pref-ORG), which generates critique suggestions in addition to recommendations according to users' preferences. The critique is a form of feedback (“I would like something cheaper than this one”) that users can provide to the currently displayed product, with which the system may better predict what the user truly wants. We compare the preference-based organization technique with related approaches, including the ones that also produce critique candidates, but without the consideration of user preferences. A simulation setup is first presented, that identified Pref-ORG's significantly higher algorithm accuracy in predicting critiques and choices that users should intend to make, followed by a real-user evaluation which practically verified its significant impact on saving users' decision effort. <s> BIB007
Providing recommendations that satisfy the criteria listed above is not enough for effective result display. In addition, it is important to assist users in understanding why these items are presented to them. To date, many researchers have demonstrated that providing good explanations for recommendations could help inspire users' trust and satisfaction, increase users' involvement and educate users on the internal logic of the system BIB001 Swearingen 2001, 2002; BIB003 Tintarev and Masthoff 2007a,b) . For instance, BIB001 discussed the concept of explanation for automated collaborative filtering systems, such as how the explanation should be implemented based on users' conceptual model of the recommending process and whether providing explanations could improve users' acceptance of a system. They highlighted that the benefits of explanation interface are justification (users could decide how much confidence to place), user involvement (allowing users to add their knowledge to complete the decision process), education (educating users about the computational processes used in generating recommendations), and acceptance (improving the acceptance of a recommender system because it helps users make a decision). BIB002 found that users like transparent recommendations. The more they have to "pay", the more transparency they require. They stated that the recipient of a recommendation has a number of ways to decide whether to trust the recommendation: scrutinizing the similarity between the taste of the recipient and the recommender; assessing the success of prior suggestions from the recommender; and asking the recommender for more information about why the recommendation was made. Similarly, recommender systems need to offer the user some ways to judge the appropriateness of recommendations. Understanding the relationship between the system input (e.g., ratings made by user) and output (recommendations) allows the user to initiate a predictable and efficient interaction with the system. As we indicated previously, perceived accuracy of a recommendation is also dependent on whether or not the user sees a correspondence between the preferences they express and the recommendation presented by the system BIB003 . Grounded on prior works on explanation, BIB005 concluded several possible goals of explanations, including transparency, scrutability, trust, effectiveness, persuasiveness, efficiency, and satisfaction. Transparency explains how the system works, i.e. how a recommendation was chosen. Scrutability allows users to correct reasoning and system assumptions where needed. Trust could increase users' confidence in the system. A user may be more forgiving if they understand why a bad recommendation has been made, and hence prevent it from occurring again. Persuasiveness is the ability to convince users to try or buy. It is important to consider that too much persuasion may backfire if certain users realize that they have bought items that they do not really want. Effectiveness is the ability of a system to assist the user in making accurate decision about which recommendations to utilize. Efficiency is the ability that explanations may make it faster for users to decide which recommended item is the best for them. Efficiency may be improved by allowing the user to understand the relation between competing options. Satisfaction is a quality that makes the use of the system fun. Several papers also reported user studies to determine effective explanation mechanisms. BIB001 performed experiments on the use of 23 different graphical displays to "explain" why each recommendation was given. They suggested that there are multiple ways for the system to convey its inner logic to the user: an explanation (e.g. "this item was recommended to you because you rated 'x' positively"); predicted ratings (e.g. "we think you'll give this item an 8.5 out of 10"); including a few familiar recommendations (i.e., ones from artists or writers who are close to input items); community opinions (showing both reviews and numerical ratings). BIB004 presented an organization-based explanation interface where the best matching item is displayed at the top of the interface along with several categories of other recommended products (see Fig. 3 ). Each category is labeled with a title explaining the characteristics of the items the respective category contains. In order to understand whether the organization interface is a more effective way to explain recommendations, an empirical study was conducted to compare the organization interface with the traditional explanation view (where each item includes an explanation), in a within-subjects design. A total of 72 volunteers (19 females) were recruited as participants in the user study. The results showed that the organization interface significantly increases users' perception of competence, inspires their trust, and enhances their intention to save cognitive effort and use the interface again in the future. Moreover, the study found that the actual time spent looking for a product did not have a significant impact on users' subjective perceptions. This indicates that less time spent on the interface, while important in reducing decision effort, cannot be used alone in predicting what users may subjectively experience. Based on these empirical findings, the authors concluded a set of principles for the effective design of organization interfaces. More details (including the algorithm steps of generating the content for such interfaces) can be referred to in BIB007 . On the other hand, user-generated social contents, such as tags, have also been exploited to generate explanations. One recent method by BIB006 developed Fig. 3 The organization-based explanation interface to inspire user trust in recommendations BIB004 tagsplanation to show tags along with a recommended item. The displayed tags are most relevant to the item and also most likely preferred by the user. They integrated the approach in a rating-based recommender and identified that this tag-based explanation can help users understand why the item was recommended, decide if they like the item, and determine if the item fits their current mood. In recent years, commercial websites have also realized the importance of explanations. For example, features such as "why this was recommended" have appeared on Netflix, Amazon and Pandora. The corresponding guideline is hence: Guideline 19 Consider explaining why the system recommends the suggested items. These aspects can be highly correlated to users' satisfaction, sense of control, and trust-inspired behavior intentions, such as the intention to save effort and the intention to return.