DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Content and Collaboration-based Modelling <s> The limitations of traditional knowledge representation methods for modeling complex human behaviour led to the investigation of statistical models. Predictive statistical models enable the anticipation of certain aspects of human behaviour, such as goals, actions and preferences. In this paper, we motivate the development of these models in the context of the user modeling enterprise. We then review the two main approaches to predictive statistical modeling, content-based and collaborative, and discuss the main techniques used to develop predictive statistical models. We also consider the evaluation requirements of these models in the user modeling context, and propose topics for future research. <s> BIB001 </s> DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Content and Collaboration-based Modelling <s> Recommender systems represent user preferences for the purpose of suggesting items to purchase or examine. They have become fundamental applications in electronic commerce and information access, providing suggestions that effectively prune large information spaces so that users are directed toward those items that best meet their needs and preferences. A variety of techniques have been proposed for performing recommendation, including content-based, collaborative, knowledge-based and other techniques. To improve performance, these methods have sometimes been combined in hybrid recommenders. This paper surveys the landscape of actual and possible hybrid recommenders, and introduces a novel hybrid, EntreeC, a system that combines knowledge-based recommendation and collaborative filtering to recommend restaurants. Further, we show that semantic ratings obtained from the knowledge-based part of the system enhance the effectiveness of collaborative filtering. <s> BIB002 </s> DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Content and Collaboration-based Modelling <s> The huge amount of information available in the currently evolving world wide information infrastructure at any one time can easily overwhelm end-users. One way to address the information explosion is to use an ’information filtering agent‘ which can select information according to the interest and/or need of an end-user. However, at present few information filtering agents exist for the evolving world wide multimedia information infrastructure. In this study, we evaluate the use of feature-based approaches to user modeling with the purpose of creating a filtering agent for the video-on-demand application. We evaluate several feature and clique-based models for 10 voluntary subjects who provided ratings for the movies. Our preliminary results suggest that feature-based selection can be a useful tool to recommend movies according to the taste of the user and can be as effective as a movie rating expert. We compare our feature-based approach with a clique-based approach, which has advantages where information from other users is available. <s> BIB003 </s> DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Content and Collaboration-based Modelling <s> The Lumiere Project centers on harnessing probability and utility to provide assistance to computer software users. We review work on Bayesian user models that can be employed to infer a user's needs by considering a user's background, actions, and queries. Several problems were tackled in Lumiere research, including (1) the construction of Bayesian models for reasoning about the time-varying goals of computer users from their observed actions and queries, (2) gaining access to a stream of events from software applications, (3) developing a language for transforming system events into observational variables represented in Bayesian user models, (4) developing persistent profiles to capture changes in a user's expertise, and (5) the development of an overall architecture for an intelligent user interface. Lumiere prototypes served as the basis for the Ofice Assistant in the Microsoft Office '97 suite of productivity applications. <s> BIB004
Research on predictive statistical user modelling commonly distinguishes between content-based and collaboration-based forms of modelling BIB001 ). While content-based approaches use the user's history of actions to predict future actions, collaboration-based approaches predict the user's next action based on the actions taken by other users in the same state. Content-based techniques are typically employed when a sufficient amount of data describing the user's past behaviour is available and it appears to be a reliable indicator of his future behaviour. "Persistence of interest" is important for content-based systems. In contrast, collaboration-based approaches are attractive when large amounts of data are available from other previous users and when user behaviour is sufficiently homogeneous for actions taken by users in the same state to be likely actions of the new user. Collaboration-based systems are less influenced by long-term information and exploit better context-dependent data BIB002  . Despite being a common classification for statistical user modelling techniques, the concepts of content-based and collaboration-based learning have so far not been used in the context of user modelling for simulation-based strategy learning. Nevertheless, it is interesting to consider the problem of modelling user dialogue behaviour in the light of the above classification. Indeed, all of the modelling techniques which are currently available for simulation-based learning are primarily collaborative approaches. This is not surprising given that we typically have a fair amount of dialogue data from many different previous users but only a small amount of data per user. However, many characteristics of individual user behaviour (such as expertise and cooperativeness) are not covered well by the currently available techniques. As we will further discuss in the section on user model evaluation, the current user models generate a variety of user behaviour which reflects the statistical distribution in the training data, but they do not acknowledge that some aspects of user behaviour are likely to remain fixed over the course of a dialogue while others are likely to be variable. It is hence desirable to combine the collaborative models currently available with content-based modelling components to ensure that user actions are taken in accordance with the previous user dialogue history and that they reflect individual user characteristics. The models presented by Scheffler and Young and by Pietquin (reviewed above) both introduce the concept of a user goal in their user models to achieve goal-consistency over the course of a dialogue. Pietquin further introduces a simple model of user memory and suggests further work on modelling user satisfaction. As we have discussed in the previous section, the use of such content-based forms of modelling in otherwise collaborative models is clearly beneficial to the realism of the simulated user behaviour. We argue that user dialogue behaviour is in fact a prime candidate for combining content-based and collaboration-based approaches. It is interesting to note that it is well known in the user modelling community that collaboration-based approaches naturally tend to model an average user unless careful steps are taken to model individual users or user groups (such as experts and novices, for instance) BIB003 BIB004 . It is only in very recent evaluation efforts  ) that this pitfall has been highlighted in research on strategy learning. As Zukerman and Albrecht BIB001 formulate: "the fundamental inability of the collaborative approach to represent the idiosyncrasies of individual users calls for a solution which combines both (content-based and collaboration based) types of modelling approaches." We believe this to be equally true for user modelling in the context of dialogue strategy learning.