Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Diversity <s> In this work we present topic diversification, a novel method designed to balance and diversify personalized recommendation lists in order to reflect the user's complete spectrum of interests. Though being detrimental to average accuracy, we show that our method improves user satisfaction with recommendation lists, in particular for lists generated using the common item-based collaborative filtering algorithm.Our work builds upon prior research on recommender systems, looking at properties of recommendation lists as entities in their own right rather than specifically focusing on the accuracy of individual recommendations. We introduce the intra-list similarity metric to assess the topical diversity of recommendation lists and the topic diversification approach for decreasing the intra-list similarity. We evaluate our method using book recommendation data, including offline analysis on 361, !, 349 ratings and an online study involving more than 2, !, 100 subjects. <s> BIB001 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Diversity <s> Recommender systems have shown great potential to help users find interesting and relevant items from within a large information space. Most research up to this point has focused on improving the accuracy of recommender systems. We believe that not only has this narrow focus been misguided, but has even been detrimental to the field. The recommendations that are most accurate according to the standard metrics are sometimes not the recommendations that are most useful to users. In this paper, we propose informal arguments that the recommender community should move beyond the conventional accuracy metrics and their associated experimental methodologies. We propose new user-centric directions for evaluating recommender systems. <s> BIB002 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Diversity <s> Two music recommender websites, Pandora (a content-based recommender) and Last.fm (a rating-based social recommender), were compared side-by-side in a within-subject user study involving 64 participants. The main objective was to investigate users’ initial adoption of recommender technology and their subjective perception of the respective systems. Results show that a simple interface design, the requirement of less initial effort, and the quality of recommended items (accuracy, novelty and enjoyability) are some of the key design features that such websites rely on to break the initial entrance barrier in becoming a popular website. <s> BIB003 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Diversity <s> Critiquing-based recommenders do not require users to state all of their preferences upfront or rate a set of previously experienced products. Compared to other types of recommenders, they require relatively little user effort, especially initially, despite potential accuracy problems. On the other hand, they rely on a set of critiques to elicit users feedback in order to improve accuracy. Thus the better the critiques are, the more accurately and efficiently the system becomes in generating its recommendations. This method has been successfully applied to high-involvement products. However, it was never tested on public taste products such as music, films, perfumes, fashion goods or wine. Indeed our initial trial adapting traditional critiquing methods to this new domain led to unsatisfactory results. This has motivated us to develop a novel approach named "editorial picked critiques" (EPC) that accounts for users' needs for popularity information, editorial suggestions, as well as their needs for personalization and diversity. Through an empirical study, we demonstrate that EPC presents a viable recommender approach and is superior on several dimensions to critiques generated by data mining methods. <s> BIB004 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Diversity <s> Even though people are attracted by large, high quality recommendation sets, psychological research on choice overload shows that choosing an item from recommendation sets containing many attractive items can be a very difficult task. A web-based user experiment using a matrix factorization algorithm applied to the MovieLens dataset was used to investigate the effect of recommendation set size (5 or 20 items) and set quality (low or high) on perceived variety, recommendation set attractiveness, choice difficulty and satisfaction with the chosen item. The results show that larger sets containing only good items do not necessarily result in higher choice satisfaction compared to smaller sets, as the increased recommendation set attractiveness is counteracted by the increased difficulty of choosing from these sets. These findings were supported by behavioral measurements revealing intensified information search and increased acquisition times for these large attractive sets. Important implications of these findings for the design of recommender system user interfaces will be discussed. <s> BIB005 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Diversity <s> Research increasingly indicates that accuracy cannot be the sole criteria in creating a satisfying recommender from the users' point of view. Other criteria, such as diversity, are emerging as important characteristics for consideration as well. In this paper, we try to address the problem of augmenting users' perception of recommendation diversity by applying an organization interface design method to the commonly used list interface. An in-depth user study was conducted to compare an organization interface with a standard list interface. Our results show that the organization interface indeed effectively increased users' perceived diversity of recommendations, especially perceived categorical diversity. Furthermore, 65% of users preferred the organization interface, versus 20% for the list interface. 70% of users thought the organization interface is better at helping them perceive recommendation diversity versus only 15% for the list interface. <s> BIB006 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Diversity <s> Research on recommender systems typically focuses on the accuracy of prediction algorithms. Because accuracy only partially constitutes the user experience of a recommender system, this paper proposes a framework that takes a user-centric approach to recommender system evaluation. The framework links objective system aspects to objective user behavior through a series of perceptual and evaluative constructs (called subjective system aspects and experience, respectively). Furthermore, it incorporates the influence of personal and situational characteristics on the user experience. This paper reviews how current literature maps to the framework and identifies several gaps in existing work. Consequently, the framework is validated with four field trials and two controlled experiments and analyzed using Structural Equation Modeling. The results of these studies show that subjective system aspects and experience variables are invaluable in explaining why and how the user experience of recommender systems comes about. In all studies we observe that perceptions of recommendation quality and/or variety are important mediators in predicting the effects of objective system aspects on the three components of user experience: process (e.g. perceived effort, difficulty), system (e.g. perceived system effectiveness) and outcome (e.g. choice satisfaction). Furthermore, we find that these subjective aspects have strong and sometimes interesting behavioral correlates (e.g. reduced browsing indicates higher system effectiveness). They also show several tradeoffs between system aspects and personal and situational characteristics (e.g. the amount of preference feedback users provide is a tradeoff between perceived system usefulness and privacy concerns). These results, as well as the validated framework itself, provide a platform for future research on the user-centric evaluation of recommender systems. <s> BIB007
Recent research work suggests that the set of recommended items should maintain a certain level of diversity, even at the risk of compromising its overall accuracy. For example, if users only get a recommendation list full of albums by the same artist, the diversity level in this list is low. BIB003 pointed out that low diversity could disappoint users and decrease their decision confidence. BIB002 noticed the same problem and stated that the item-to-item collaborative filtering algorithm may trap users in a "similarity hole", only giving exceptionally similar recommendations. Thus, in recent years, more researchers have focused on generating diverse recommendations and aiming to reach an optimal balance between diversity and similarity ( (2003) highlighted the pitfalls of naively incorporating diversity-enhancing techniques into existing recommender systems. They pointed out that diversity should be provided adaptively rather than being enhanced in each and every recommender cycle. If a recommender system appears to be close to the target item, then diversity should be limited to avoid missing this item. But if the recommender system is not correctly focused, diversity can be used to help refocus more effectively. BIB001 proposed a topic diversification approach based on taxonomy-based similarity. They compared not only the accuracy measures in different levels of diversification for both user-based and item-based CF, but also subjective satisfaction results from a large scale user survey. Their results showed that users' overall satisfaction with recommendation lists goes beyond accuracy and involves other factors, e.g., the diversity of the result set. In the same study, they found that human perception seems to capture only a certain level of diversification inherent to a list. Beyond that point, it is difficult for users to notice the increasing diversity. In another words, they suggest that users' diversity perception may not correlate to the diversity controlled by algorithms. BIB005 reported an experiment aimed at understanding if objective diversity has a straightforward effect on perceived diversity. After rating 10 movies from the MovieLens dataset, users were told to select a movie to watch and then answer a satisfaction questionnaire. The 137 subjects were divided into 3 × 3 conditions, independently varying the underlying recommendation algorithms (Most Popular, kNN, Matrix Factorization) and the degree of diversifications (none, little, a lot). Objective diversification was implemented based on movie genre, using a greedy search method proposed by BIB001 . The main outcomes from this user study seem to confirm that users do not necessarily perceive the diversified recommendations as more diverse. In BIB004 , an interface design method was described to increase users' perception of recommendation diversity by grouping and categorizing any result set into an organization interface. The method, called the Editorial Picked Critiques (EPC), creates up to seven category titles such as "more popular and cheaper", "same brand and cheaper", "just as popular but cheaper" and put items into the right group. BIB006 conducted an in-depth user study to compare the organization interface with the standard list interface, while keeping the recommendation contents identical. The results show that the organization interface indeed effectively increased users' perceived diversity of recommendations, especially perceived categorical variety. Correlation results further show that perceived diversity more significantly influences users' perceived ease of use and usefulness of the recommender, positive attitudes toward the system and behavioral intentions. More than 65% of users prefer the organization interface, versus 20% for the list interface. 50% of users thought the organization interface was better at recommending products to them versus only 10% for the list interface. In addition to using organization interfaces, the result set size and mixing best items with low-ranked items could also affect users' perceived diversity. BIB007 conducted an experiment with 174 Dutch participants, where each user first rated 10 movies from the MovieLens dataset and was told to select a movie from a recommendation set to watch. Each user was randomly assigned to one of three conditions. In Top-5, users reviewed the best five recommendations to select a movie to watch; in Top-20, users reviewed the best 20 recommendations; and in Lin-20, users reviewed the best five recommendations, appended with items which only ranked 99, 199, 299, continuing through 1499. In other words, in Lin-20, the result set was intentionally manipulated to include some low-ranked items. Results from this study show that while a larger result set (Top-20) increases the perceived diversity of the recommended items, it also increased users' choice difficulty because more information processing is required. When the result set includes both attractive and low-ranked items, even though this approach compromised on the overall quality of the items, it increased choice satisfaction because the inferior items, when contrasted to the accurate items, provided a decision context that reduced choice difficulty. We therefore suggest: Guideline 17 Consider adding diversity to the recommendation set. Since perceived diversity is not directly correlated with predicted diversity, additional methods, such as interface design and adding low-ranked items to the set, can be considered. Perceived diversity is strongly correlated to the perceived usefulness of the system and users' satisfaction.