Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> User effort <s> Information displays influence decision processes by facilitating some decision strategies while hindering others. Component characteristics of displays, such as the form, organization, and sequence of information, influence decision processes through an adaptive mechanism whereby a decision maker balances the desire to maximize accuracy against the desire to minimize effort. Variations in the information display lead to changes in the anticipated effort and anticipated accuracy of each available strategy and, therefore, provide an incentive for decision makers to use different decision processes. Research in this area can provide guidance regarding the use of displays and other decision-aiding approaches. <s> BIB001 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> User effort <s> Recommender systems have become valuable resources for users seeking intelligent ways to search through the enormous volume of information available to them. One crucial unsolved problem for recommender systems is how best to learn about a new user. In this paper we study six techniques that collaborative filtering recommender systems can use to learn about new users. These techniques select a sequence of items for the collaborative filtering system to present to each new user for rating. The techniques include the use of information theory to select the items that will give the most value to the recommender system, aggregate statistics to select the items the user is most likely to have an opinion about, balanced techniques that seek to maximize the expected number of bits learned per presented item, and personalized techniques that predict which items a user will have an opinion about. We study the techniques thru offline experiments with a large pre-existing user data set, and thru a live experiment with over 300 users. We show that the choice of learning technique significantly affects the user experience, in both the user effort and the accuracy of the resulting predictions. <s> BIB002 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> User effort <s> The internet presents people with an increasingly bewildering variety of choices. Online consumers have to rely on computerized search tools to find the most preferred option in a reasonable amount of time. Recommender systems address this problem by searching for options based on a model of the user's preferences. We consider example critiquing as a methodology for mixed-initiative recommender systems. In this technique, users volunteer their preferences as critiques on examples. It is thus important to stimulate their preference expression by selecting the proper examples, called suggestions. We describe the look-ahead principle for suggestions and describe several suggestion strategies based on it. We compare them in simulations and, for the first time, report a set of user studies which prove their effectiveness in increasing users' decision accuracy by up to 75%. <s> BIB003 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> User effort <s> Two music recommender websites, Pandora (a content-based recommender) and Last.fm (a rating-based social recommender), were compared side-by-side in a within-subject user study involving 64 participants. The main objective was to investigate users’ initial adoption of recommender technology and their subjective perception of the respective systems. Results show that a simple interface design, the requirement of less initial effort, and the quality of recommended items (accuracy, novelty and enjoyability) are some of the key design features that such websites rely on to break the initial entrance barrier in becoming a popular website. <s> BIB004 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> User effort <s> Despite the explosive growth of electronic commerce and the rapidly increasing number of consumers who use interactive media such as the World Wide Web for prepurchase information search and online shopping, very little is known about how consumers make purchase decisions in such settings. A unique characteristic of online shopping environments is that they allow vendors to create retail interfaces with highly interactive features. One desirable form of interactivity from a consumer perspective is the implementation of sophisticated tools to assist shoppers in their purchase decisions by customizing the electronic shopping environment to their individual preferences. The availability of such tools, which we refer to as interactive decision aids for consumers, may lead to a transformation of the way in which shoppers search for product information and make purchase decisions. The primary objective of this paper is to investigate the nature of the effects that interactive decision aids may have on consumer decision making in online shopping environments. ::: ::: While making purchase decisions, consumers are often unable to evaluate all available alternatives in great depth and, thus, tend to use two-stage processes to reach their decisions. At the first stage, consumers typically screen a large set of available products and identify a subset of the most promising alternatives. Subsequently, they evaluate the latter in more depth, perform relative comparisons across products on important attributes, and make a purchase decision. Given the different tasks to be performed in such a two-stage process, interactive tools that provide support to consumers in the following respects are particularly valuable: 1 the initial screening of available products to determine which ones are worth considering further, and 2 the in-depth comparison of selected products before making the actual purchase decision. This paper examines the effects of two decision aids, each designed to assist consumers in performing one of the above tasks, on purchase decision making in an online store. ::: ::: The first interactive tool, a recommendation agent RA, allows consumers to more efficiently screen the potentially very large set of alternatives available in an online shopping environment. Based on self-explicated information about a consumer's own utility function attribute importance weights and minimum acceptable attribute levels, the RA generates a personalized list of recommended alternatives. The second decision aid, a comparison matrix CM, is designed to help consumers make in-depth comparisons among selected alternatives. The CM allows consumers to organize attribute information about multiple products in an alternatives × attributes matrix and to have alternatives sorted by any attribute. ::: ::: Based on theoretical and empirical work in marketing, judgment and decision making, psychology, and decision support systems, we develop a set of hypotheses pertaining to the effects of these two decision aids on various aspects of consumer decision making. In particular, we focus on how use of the RA and CM affects consumers' search for product information, the size and quality of their consideration sets, and the quality of their purchase decisions in an online shopping environment. ::: ::: A controlled experiment using a simulated online store was conducted to test the hypotheses. The results indicate that both interactive decision aids have a substantial impact on consumer decision making. As predicted, use of the RA reduces consumers' search effort for product information, decreases the size but increases the quality of their consideration sets, and improves the quality of their purchase decisions. Use of the CM also leads to a decrease in the size but an increase in the quality of consumers' consideration sets, and has a favorable effect on some indicators of decision quality. ::: ::: In sum, our findings suggest that interactive tools designed to assist consumers in the initial screening of available alternatives and to facilitate in-depth comparisons among selected alternatives in an online shopping environment may have strong favorable effects on both the quality and the efficiency of purchase decisions-shoppers can make much better decisions while expending substantially less effort. This suggests that interactive decision aids have the potential to drastically transform the way in which consumers search for product information and make purchase decisions. <s> BIB005 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> User effort <s> The World Wide Web has become an important source of information for tourists planning their vacation. So, destination recommendation systems supporting users in their decision making process by suggesting suitable holiday destinations or packages based on user profiles are a vivid area of research. Considering the complex and often tedious task to obtain such profiles we are exploring a new direction to manufacture user profiles. Having in mind that a picture paints a thousand words we conducted an online survey that allows investigating the relationship between tourism-related photographs and tourist types. In a nutshell, our findings show a significant relationship between different tourist types and the preference for particular visual impressions conveyed by photographs. Thus, tourist types can be determined by representative photos without necessarily requesting users to provide additional information. <s> BIB006 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> User effort <s> Recommender systems try to address the "new user problem" by quickly and painlessly learning user preferences so that users can begin receiving recommendations as soon as possible. We take an expanded perspective on the new user experience, seeing it as an opportunity to elicit valuable contributions to the community and shape subsequent user behavior. We conducted a field experiment in MovieLens where we imposed additional work on new users: not only did they have to rate movies, they also had to enter varying numbers of tags. While requiring more work led to fewer users completing the entry process, the benefits were significant: the remaining users produced a large volume of tags initially, and continued to enter tags at a much higher rate than a control group. Further, their rating behavior was not depressed. Our results suggest that careful design of the initial user experience can lead to significant benefits for an online community. <s> BIB007 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> User effort <s> Conversational recommender systems support a structured human-computer interaction in order to assist online tourists in important online activities such as travel planning and dynamic packaging. In this paper we describe the effects and advantages of a novel recommendation methodology based on Machine Learning techniques. It allows conversational systems to autonomously improve an initial strategy in order to learn a new one that is more effective and efficient. We applied and tested our approach within a prototype of an online travel recommender system in collaboration with the Austrian Tourism portal (Austria.info). In this paper, we present the features of this technology and the results of the online evaluation. We show that the learned strategy adapts its actions to the served users, and deviates from a rigid initial strategy. More importantly, we show that the optimal strategy is able to assist online tourists in acquiring their goals more efficiently than the initial strategy. It can be used by the system designer to understand the limitations of an existing interaction design and guide him in the adoption of a new one that is capable to improve customer relationship, the usage of their web site, and the conversion rate of their online users. <s> BIB008 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> User effort <s> Our previous research indicates that using personality quizzes is a viable and promising way to build user profiles to recommend entertainment products Based on these findings, our current research further investigates the feasibility of using personality quizzes to build user profiles not only for an active user but also his or her friends We first propose a general method that infers users' music preferences in terms of their personalities Our in-depth user studies show that while active users perceive the recommended items to be more accurate for their friends, they enjoy more using personality quiz based recommenders for finding items for themselves Additionally, we explore if domain knowledge has an influence on users' perception of the system We found that novice users, who are less knowledgeable about music, generally appreciated more personality-based recommenders Finally, we propose some design issues for recommender systems using personality quizzes. <s> BIB009
Effort expenditure has been a central subject of study in decision theories and lately in recommender systems. This attribute concerns the time and effort users expend interacting with the RS before they receive decision assistance. While users cannot always perceive the system's usefulness in the beginning, they are confronted with the effort requirement at the onset of RS use. The central issue involves determining how much effort users are willing to invest in RS use, and how RS designers can best manage this cost-benefit tradeoff. In earlier rating-based recommender systems, users were asked to rate up to 50 items before receiving any recommendations. While such an elaborate preference elicitation is likely to help the system more accurately predict what may interest users, such a painstaking process may unintentionally drive them away. BIB007 analyzed historical data over 5,000 users who have had at least 65 ratings with MovieLens. The study compared the recommender system's prediction accuracy for these users when they had only rated 1 movie up to 15 movies. The accuracy was measured by offline experiments that computed the Mean Absolute Error (MAE) of system prediction. As expected, as users rated an increasingly larger number of movies, the prediction error decreased, resembling an inverse curve, asymptotically approaching the limit of 0.65. In other words, while users were able to increasingly obtain more precise recommendations by rating more movies, once the asymptotic line is reached, their subsequent effort does not result in any significant reward. In fact, asking users to provide 5 ratings seems to be the optimum for this accuracy versus effort tradeoff. Going from 1 to 5 movie ratings, users were able to reach a significant increase in accuracy (from 0.94 to 0.76 in MAE, a decrease of 20% in prediction error). On the other hand, users who gave 15 ratings were only able to reduce prediction errors to 0.687 in MAE. BIB004 compared two music recommender systems in a controlled study involving 64 real users, in a with-in subject experiment design. This setting allows the users to experience two systems one after the other, immediately or within few days' time. It provides more statistical power as the sample size doubles (more data points collected), and, more importantly, it is possible to assess their preference between the two systems. The disadvantage is that user fatigue and participation motivation need to be carefully managed. In this study, users were able to listen to a series of recommended songs shortly after interacting with one system, by specifying the title of one song, the name of an artist, or the name of an album. In the other system, users had to install a plug-in to initialize their music profiles and wait for few days (∼5 days) before receiving personalized recommendations. The results show more than 70% of the tested users preferred the first system whose time-to-recommendation is almost instantaneous over a system which requires users to reveal their preferences over several days. General satisfaction for the first system is 4.1 versus only 3.4 for the second system, a decrease of more than 20% on the Likert scale.  conducted an empirical study in which a simulated shopbot was used to evaluate users' subjective opinions. A key variable in the study was the shopbot's response time. The study found that users' preference for the shopping agent decreased when the time-to-recommendation increased. As the field matured, it became apparent that placing more emphasis on the algorithm's accuracy than on users' effort would cause serious user experience problems. Today it is no longer a common practice to elicit 50 ratings from users or extensively request their preferences BIB003 . For example, MovieLens requires 15 ratings in the registration stage; Jinni (http://www.jinni.com) and Filmaffinity (http:// www.filmaffinity.com) ask users to rate around 10 movies to obtain recommendations. If other methods are used, e.g., personality quizzes BIB009 , the initial effort should also be limited to the range of 10-15 questions. According to behavioral decision theories, human decision makers focus more on reducing cognitive effort than on improving decision accuracy, because feedback on effort expenditure tends to be immediate while feedback on accuracy is subject to delay and ambiguity  BIB001 BIB005 . This infers that the difference in accuracy among recommender engines might be less important than the difference in their demands for user effort. Thus, we suggest: Guideline 1 Minimize preference elicitation in profile initialization. In the tradeoff between accuracy versus effort, users are likely to settle on the immediate benefit of saving effort over delayed gratification of higher accuracy. This guideline can be applied to all types of recommender systems that require an explicit preference elicitation process. While it is a general trend that users avoid effort, not all of them have the same predisposed effort levels. It is possible to design an adaptive system that tailors the user interaction effort to his or her individual characteristics. It is also imaginable that methods differ in terms of how they motivate users to expend this effort. BIB008 proposes treating the system-user interaction design with adaptive recommendation strategies, dynamically selecting the most optimal method as the system learns about the characteristics of an active user. In particular, after a user makes a request (e.g., a search for hotels), the system determines the kinds of recommending computations it will perform and the specific user interface it will show. To select an appropriate action to execute given the user's current situation, they applied reinforcement learning techniques to decide on the optimal policies. With the involvement of 544 users, they tested the adaptive learning approach within an online travel recommender and validated that this method enables more users to add products to their carts with less effort (i.e., reduced session duration and reduced number of queries). BIB006 examined an innovative technique to elicit user preferences for vacation destinations based on photos selected by users. An online survey of over 476 subjects was conducted, where each user selected photos from a catalog of 60 photos classified into 17 types such as Anthropologist, Escapist, Archaeologist, etc. Correspondence analysis was then used to produce a mapping of relationships between tourist types and the photographs. This work showed that preference elicitation is not restricted to methods such as rating items. It can be obtained from asking users to choose visual impressions. Further, it seems that users can answer the visual query over 60 photo items without much effort, thus suggesting that a more entertaining process can increase users' predisposed effort levels. According to these two investigations, we suggest: Guideline 2 Consider crafting the initial elicitation process adaptive to individuals' characteristics, such as their expertise and knowledge of the domain, and consider using more entertaining means to motivate users to provide input. This guideline has been tested in the travel product domain and has the potential to be applicable to all recommenders. 3.2 Composition of items to rate BIB002 studied six elicitation strategies: pure random, MovieLens Classique, popularity, pure entropy, combination of popularity and entropy, and itemto-item personalization. Random selects items to present arbitrarily with uniform probability over the universe of items. MoviesLens Classique selects one movie randomly from an ad hoc list of popular movies, with the rest on that page randomly taken from all movies. Popularity lists the movies in descending order of number ratings, with the highest rated movies displayed first. The pure entropy strategy proposes items with higher information gains (i.e., movies that some users hate and others like). The combination of popularity and entropy (Pop * Ent) strategy is a hybrid approach that combines the popularity and pure entropy methods. The reasoning was to make the two strategies compensate for each other, in order to address the popularity strategy's lack of information strength and the entropy strategy's likelihood of making users rate a movie that they have not seen. The item-to-item personalized method learns which items users have rated thus far and then suggests some related ones for them to rate next. In an experiment where historical data was used, researchers mimicked an online sign-up process. They used each strategy to present a total of 30, 45, 60, or 90 movies to each "user". The experiment showed that the item-to-item strategy performed the best in picking movies "users" could rate, while Pure Entropy did the worst. Specifically, when presented with 30 movies, simulated "users" (for whom the database has the ratings) were able to rate up to 20 movies with the item-to-item strategy and up to 15 movies with the popularity and Pop * Ent methods. However, using the Pure Entropy list, less than 2 could be rated. Because the Pure Entropy strategy turned out to be an inferior method, it was eliminated in the follow-up online study. In the subsequent online user study, 351 new users took part in the sign-up process during a 10-day period (49 users left during the process). The system continuously presented a set of movies until users voluntarily rated at least ten of them. The results indicate that users in the popularity group rated 10 items within an average of 1.9 pages, while the item-to-item group went through 2.3 pages on average to rate 10 items. Log Pop * Ent, a variant of Pop * Ent, required much more effort from users, as the average number of pages was 4.0 for rating 10 movies. The MovieLens Classique method involved even more effort, requiring 7.0 pages before users rated 10 movies. This user study hence suggests that the popularity and item-to-item strategies are by far the most effective methods with over 90% of users being able to sign up in five pages or less. Based on these results, we are able to draw a guideline for rating-based recommender systems: Guideline 3 Consider showing a mixture of popular and personalized items for users to rate when eliciting their preferences. The relative proportion of the two strategies depends on the product domains; for some products (e.g., restaurants, accommodations) users are less likely to have experienced popular items, so the proportion of personalized items should be higher.