Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Critiquing support <s> A new kind of user interface for information retrieval has been designed and implemented to aid users in formulating a query. The system, called RABBIT, relies upon a new paradigm for retrieval by reformulation , based on a psychological theory of human remembering. The paradigm actually evolved from an explicit attempt to design a 'natural' interface which imitated human retrieval processes. To make a query in RABBIT, the user interactively refines partial descriptions of his target item(s) by criticizing successive example (and counterexample) instances that satisfy the current partial description. Instances from the database are presented to the user from a perspective inferred from the user's query description and the structure of the knowledge base. Among other things, this constructed perspective reminds users of likely terms to use in their descriptions, enhances their understanding of the meaning of given terms, and prevents them from creating certain classes of semantically improper query descriptions. RABBIT particularily facilitates users who approach a database with only a vague idea of what it is that they want and who thus, need to be guided in the (re)formulation of their queries. RABBIT is also of substantial value to casual users who have limited knowledge of a given database or who must deal with a multitude of databases. <s> BIB001 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Critiquing support <s> While the explosion of online information has introduced new opportunities for finding and using electronic data, it has also underscored the problem of isolating useful information and making sense of large, multidimensional information spaces. In response to this problem, we have developed an approach to building data tour guides, called FindMe systems. These programs know enough about an information space to help users navigate through it, making sure they not only come away with useful information but also insights into the structure of the information space itself. In these systems, we have combined the idea of instance-based browsing, which involves structuring retrieval around the critiquing of previously retrieved examples, and retrieval strategies, or knowledge-based heuristics for finding relevant information. This article illustrates these techniques with examples of working FindMe systems, and describes the similarities and differences between them. FindMe tour guides help users to select the perfect car, movie, restaurant, stereo or apartment on the World Wide Web. <s> BIB002 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Critiquing support <s> When it comes to buying expensive goods people expect to be skillfully steered through the options by well-informed sales assistants that are capable of balancing the user’s many and varied requirements. In addition users often need to be educated about the product-space, especially if they are to come to understand what is available and why certain options are being recommended by the sales-assistant. The same issues arise in interactive recommender systems, our online equivalent of a sales assistant and explanation in recommender systems, as a means to educate users and justify recommendations, is now well accepted. In this paper we focus on a novel approach to explanation. Instead of attempting to justify a particular recommendation we focus on how explanations can help users to understand the recommendation opportunities that remain if the current recommendation should not meet their requirements. We describe how this approach to explanation is tightly coupled with the generation of compound critiques, which act as a form of feedback for the users. And we argue that these explanation-rich critiques have the potential to dramatically improve recommender performance and usability. <s> BIB003 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Critiquing support <s> We consider constructive preference elicitation for decision aid systems in applications such as configuration or electronic catalogs. We are particularly interested in supporting decision tradeoff, where preferences are revised in response to the available outcomes. In several user-involved decision aid systems we designed in the past, we were able to observe three generic tradeoff strategies that people like to use. We show how a preference model based on soft constraints is well-suited for supporting these strategies. Such a framework provides an agile preference model particularly powerful for preference revision during tradeoff analysis. We further show how to integrate the constraint-based preference model with an interaction model called example-critiquing. We report on user studies which show that this model offers significant advantages over the commonly used ranked-list model, especially when the decision problem becomes complex. <s> BIB004 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Critiquing support <s> A crucial element in consumer electronic commerce is a catalog tool that not only finds the product for the user, but also convinces him that he has made the best choice. To do that, it is important to show him ample choices while keeping his interaction effort below an acceptable limit. Among the various interaction models used in operational e-commerce sites, ranked lists are by far the most popular tool for product navigation and selection. However, as the number of product features and the complexity of user's criteria increase, a ranked list's efficiency becomes less satisfactory. As an alternative, research groups from the intelligent user interface community have developed various example-based search tools, including SmartClient from our laboratory. These tools not only perform personalized search, but also support tradeoff analysis. However, despite the academic interest, example-based search paradigms have not been widely adopted in practice. We have examined the usability of such tools on a variety of tasks involving selection and tradeoff. The studies clearly show that example-based search is comparable to ranked lists on simple tasks, but significantly reduces the error rate and search time when complex tradeoffs are involved. This shows that such tools are likely to be useful particularly for extending the scope of consumer e-commerce to more complex products. <s> BIB005 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Critiquing support <s> Conversational recommender systems are commonly used to help users to navigate through complex product-spaces by alternatively making product suggestions and soliciting user feedback in order to guide subsequent suggestions. Recently, there has been a surge of interest in developing effective interfaces that support user interaction in domains of limited user expertise. Critiquing has proven to be a popular and successful user feedback mechanism in this regard, but is typically limited to the modification of single features. We review a novel approach to critiquing, dynamic critiquing, that allows users to modify multiple features simultaneously by choosing from a range of so-called compound critiques that are automatically proposed based on their current position within the product-space. In addition, we introduce the results of an important new live-user study that evaluates the practical benefits of dynamic critiquing. <s> BIB006 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Critiquing support <s> There is increasing awareness in recommender systems research of the need to make the recommendation process more transparent to users. Following a brief review of existing approaches to explanation in recommender systems, we focus in this paper on a case-based reasoning (CBR) approach to product recommendation that offers important benefits in terms of the ease with which the recommendation process can be explained and the system's recommendations can be justified. For example, recommendations based on incomplete queries can be justified on the grounds that the user's preferences with respect to attributes not mentioned in her query cannot affect the outcome. We also show how the relevance of any question the user is asked can be explained in terms of its ability to discriminate between competing cases, thus giving users a unique insight into the recommendation process. <s> BIB007 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Critiquing support <s> In a previously reported user study, we found that users were able to perform decision tradeoff tasks more efficiently and commit considerably fewer errors with the example critiquing interface than with the ranked list. We concluded that example-based search tools were likely to be useful particularly for extending the scope of consumer e-commerce to more complex products where decision making is critical. This paper presents results from a follow-up user study quantifying the benefits of tradeoff support. Users were able to refine the quality of their preference structures and improve decision accuracy by up to 57% after performing tradeoff tasks. Tradeoff support also significantly increased users' confidence in their choices. Together, these two studies show that example critiquing enables users to more accurately find what they want and be confident in their choices, while only requiring a level of effort that is comparable to the ranked list interface. <s> BIB008 </s> Evaluating recommender systems from the user’s perspective: survey of the state of the art <s> Critiquing support <s> A critiquing-based recommender system acts like an artificial salesperson. It engages users in a conversational dialog where users can provide feedback in the form of critiques to the sample items that were shown to them. The feedback, in turn, enables the system to refine its understanding of the user's preferences and prediction of what the user truly wants. The system is then able to recommend products that may better stimulate the user's interest in the next interaction cycle. In this paper, we report our extensive investigation of comparing various approaches in devising critiquing opportunities designed in these recommender systems. More specifically, we have investigated two major design elements which are necessary for a critiquing-based recommender system: critiquing coverage--one vs. multiple items that are returned during each recommendation cycle to be critiqued; and critiquing aid--system-suggested critiques (i.e., a set of critique suggestions for users to select) vs. user-initiated critiquing facility (i.e., facilitating users to create critiques on their own). Through a series of three user trials, we have measured how real-users reacted to systems with varied setups of the two elements. In particular, it was found that giving users the choice of critiquing one of multiple items (as opposed to just one) has significantly positive impacts on increasing users' decision accuracy (particularly in the first recommendation cycle) and saving their objective effort (in the later critiquing cycles). As for critiquing aids, the hybrid design with both system-suggested critiques and user-initiated critiquing support exhibits the best performance in inspiring users' decision confidence and increasing their intention to return, in comparison with the uncombined exclusive approaches. Therefore, the results from our studies shed light on the design guidelines for determining the sweetspot balancing user initiative and system support in the development of an effective and user-centric critiquing-based recommender system. <s> BIB009
In this section, we survey critiquing support techniques based on work proposed in case-based and utility-based systems in the past several years. Critiquing support essentially acts like an artificial salesperson. It engages users in a conversational dialog where users can provide feedback in the form of critiques (e.g., "I would like something cheaper") to the sample items. The feedback, in turn, enables the system to refine its understanding of the user's preferences and its prediction of what the user truly wants. In the next interaction cycle, the system is then able to recommend products that may better stimulate the user's interest. To our knowledge, this critiquing concept was first mentioned in the RABBIT system BIB001 as a new interface paradigm for formulating queries to a database. In recent years, it has evolved into two principal branches. One aims to proactively generate a set of knowledge-based critiques that users may be prepared to accept as ways to improve the current product, termed system-suggested critiques in BIB009 . This mechanism was initially used in FindMe systems BIB002 , and more recently in Dynamic Critiquing (DC) agents BIB006 . The main advantage, as detailed in related literature BIB003 BIB007 , is that system-suggested critiques not only exposes the knowledge of remaining recommendation opportunities, but also accelerates the user's critiquing process if they correspond well to the user's intended feedback criteria. An alternative critiquing mechanism does not propose pre-computed critiques, but provides a facility to stimulate users to freely create and combine critiques themselves (user-initiated critiquing support in Chen and Pu (2009)). As a typical application, the Example Critiquing (EC) agent has been developed for this goal, and its focus is on showing examples and facilitating users' composition of self-initiated critiques BIB004 BIB005 . In essence, the EC agent is capable of allowing users to choose any combinations of feature(s) to critique and the amount of variation they want to apply to the critiques. Previous work has proven that this enables users to obtain significantly higher decision accuracy, preference certainty, and sense of control, compared to non critiquing-based product search systems BIB005 BIB008 . Chen and Pu (2009) have further studied how to design effective critiquing components. Through a series of three user-trials, they identified the effects of two crucial elements: critiquing coverage (the number of items to be critiqued) and critiquing aid (the specific critiquing support), on users' decision accuracy/effort, and behavioral intentions. Specifically, the first trial (with 36 subjects) compared two typical existing applications: DC, which suggests one item during each recommendation cycle and a list of system-suggested compound critiques for users to select as improvements to the recommended item, and EC, which returns multiple items at a time and provides a user-initiated critiquing facility to assist users in freely creating their own tradeoff criteria. The results showed that EC performed significantly better than DC in improving users' objective/subjective accuracy, reducing their interaction effort and cognitive effort, and increasing their purchase and return intentions. In the second trial (with 36 new subjects), both EC and DC were modified so that their only differing factor was the critiquing aid. Although there is no significant difference between the two modified versions, participants' written comments revealed their respective strengths: system-suggested compound critiques made users feel that they obtained more knowledge about the remaining recommendation opportunities and it was easier to make critiques; a user-initiated critiquing aid allowed for detailed preference refinement and more user-control over composing users' own search criteria. Moreover, the significant effects of k recommendations in the first round (k-NIR) and k recommended items after each critiquing action (k-NCR) were identified (k = 7 in the experiments). That is, k-NIR significantly contributed to improving users' decision accuracy and trusting intentions, and k-NCR performed effectively in saving users' objective effort, including task time and critiquing cycles. In addition, it implies that this multi-item display strategy represents the key factor that led to EC's success in the first trial. Fig. 2 The hybrid-critiquing interface with both system-suggested critiques and user-initiated critiquing facility Pu 2007, 2009) The third trial (with 18 subjects) evaluated user performance in a hybrid-critiquing interface that combines both system-suggested critiques and the user-initiated critiquing facility on the same screen (see Fig. 2 ). The experiment found that users were active in making critiques using this interface. Furthermore, in comparison with user data in the system without system-suggested compound critiques or the one without user-initiated critiquing support, the hybrid system enabled users to reach significantly higher levels of decision confidence and return intentions. Based on these findings, we suggest two guidelines as follows: Guideline 12 Consider presenting several items for users to critique. This will increase their sense of control and therefore their satisfaction with the critiquing support. Guideline 13 Consider offering a hybrid-critiquing method with two types of critiquing aids: system-suggested critiques and user-initiated critiquing. The former provides more critiquing assistance, especially when the system-generated critiques closely match users' interest, while the latter provides users with a higher degree of control.