Evaluating recommender systems from the user’s perspective: survey of the state of the art <s>  <s> Based on our recent work on the development of a trust model for recommender agents and a qualitative survey, we explore the potential of building users' trust with explanation interfaces. We present the major results from the survey, which provided a roadmap identifying the most promising areas for investigating design issues for trust-inducing interfaces. We then describe a set of general principles derived from an in-depth examination of various design dimensions for constructing explanation interfaces, which most contribute to trust formation. We present results of a significant-scale user study, which indicate that the organization-based explanation is highly effective in building users' trust in the recommendation interface, with the benefit of increasing users' intention to return to the agent and save cognitive effort. <s> BIB001 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s>  <s> Research on recommender systems typically focuses on the accuracy of prediction algorithms. Because accuracy only partially constitutes the user experience of a recommender system, this paper proposes a framework that takes a user-centric approach to recommender system evaluation. The framework links objective system aspects to objective user behavior through a series of perceptual and evaluative constructs (called subjective system aspects and experience, respectively). Furthermore, it incorporates the influence of personal and situational characteristics on the user experience. This paper reviews how current literature maps to the framework and identifies several gaps in existing work. Consequently, the framework is validated with four field trials and two controlled experiments and analyzed using Structural Equation Modeling. The results of these studies show that subjective system aspects and experience variables are invaluable in explaining why and how the user experience of recommender systems comes about. In all studies we observe that perceptions of recommendation quality and/or variety are important mediators in predicting the effects of objective system aspects on the three components of user experience: process (e.g. perceived effort, difficulty), system (e.g. perceived system effectiveness) and outcome (e.g. choice satisfaction). Furthermore, we find that these subjective aspects have strong and sometimes interesting behavioral correlates (e.g. reduced browsing indicates higher system effectiveness). They also show several tradeoffs between system aspects and personal and situational characteristics (e.g. the amount of preference feedback users provide is a tradeoff between perceived system usefulness and privacy concerns). These results, as well as the validated framework itself, provide a platform for future research on the user-centric evaluation of recommender systems. <s> BIB002
While Steps 1 and 3 characterize system user interactions with a RS, Step 2 is concerned with a RS's methods and strategies for effectively presenting results to users. The key issues concern the choice of the labels and layouts for the recommended items, the internal structure of the result set, and the set composition. The recommendation label identifies the area on the screen where the recommended items are displayed. The most well-known commercial recommender systems use labels such as "Customer who viewed this also viewed", "Customers who bought this also bought", "Recommendation for you" (all by Amazon), "Movies you'll like" (by Netflix), or simply "Suggestion" (by YouTube). In addition to labels, many recent recommenders employ explanation techniques to help users understand the recommender's logic and thus augment the transparency of the system user interaction. For example, Amazon uses "because you purchased" and "because your shopping cart includes" to explain why an item was recommended to a user. In critiquing based recommenders (see detailed review in Sect. 2.2.4), the recommendation labels also explain the value tradeoffs between the user's desired item and the suggested items. Thus the labels chosen for the recommendation set play an important role in providing system transparency (as in simple explanation) and persuading users to select items recommended to them (as in tradeoff explanation). The layout for the recommendation outcome is concerned with the relative location on the computer screen for displaying the content of the result set. The right-hand longitudinal display places the items on the right hand side of the screen (as in YouTube), and the lower latitudinal display places the recommendation list at the bottom of the screen (as in Amazon). Both layout methods often display the set together with the item that the user is viewing in detail. The internal structure of the set refers to whether the set will be presented as a linear set of items (list) or whether it is further structured and organized into categories of items (as investigated in BIB001 ). The list view is the standard interface used in most commercial websites, while the latter (sometimes called the grid view) represents an emerging and novel interface in the field. Some commercial websites (e.g., www.asos.com) have started adopting this design. The set composition refers to the choice of the set size (how many items to recommend) and how the different kinds of items are mixed and balanced to make up the final set. Studies indicate that while showing one item is too few, showing more than five items increases users' choice difficulty, but increases their perception of diversity. Composing a set with a mixture of top-ranked and mediocre items reduces users' choice difficulty BIB002 . To provide a framework for the derivation of the guidelines, we define a taxonomy of user criteria for the evaluation of RS as follows: (1) the overall qualities of recommended items including its perceived accuracy, novelty, attractiveness, and diversity; (2) the ease of preference elicitation and revision processes; (3) the adequacy of layout and labels of the recommendation area; (4) its ability to assist and improve the quality of decisions users make; and (5) its ability to explain the recommended results (transparency) and inspire user trust to accept the items recommended to them. The rest of this paper is hence organized as follows. We start by presenting a basic classification of the types of recommender systems in Sect. 2. We then survey the state of the art of user experience research related to three crucial interaction aspects of a recommender: preference elicitation (Sect. 3), preference refinement (Sect. 4), and results presentation strategies (Sect. 5). For each aspect, we present key results from existing user evaluation work to derive a set of design guidelines. In preference elicitation, we are concerned with the initial interaction between the user and the recommender system. We pay particular attention to the tradeoff between accuracy and effort and draw design guidelines that ensure an optimal balance. In preference refinement and results presentation, we focus on the treatment of accuracy and user confidence. We examine various interaction techniques and develop guidelines that can help the system better motivate users to achieve high accuracy and inspire decision confidence. These sections are followed by our discussion of related works that aim at surveying and establishing an overall user evaluation framework to assess a recommender system's perceived qualities (Sect. 6). Section 7 concludes the survey.