DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Evaluation Techniques <s> Mixed-initiative systems present the challenge of finding an effective level of interaction between humans and computers. Machine learning presents a promising approach to this problem in the form of systems that automatically adapt their behavior to accommodate different users. In this paper, we present an empirical study of learning user models in an adaptive assistant for crisis scheduling. We describe the problem domain and the scheduling assistant, then present an initial formulation of the adaptive assistant's learning task and the results of a baseline study. After this, we report the results of three subsequent experiments that investigate the effects of problem reformulation and representation augmentation. The results suggest that problem reformulation leads to significantly better accuracy without sacrificing the usefulness of the learned behavior. The studies also raise several interesting issues in adaptive assistance for scheduling. <s> BIB001 </s> DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Evaluation Techniques <s> The limitations of traditional knowledge representation methods for modeling complex human behaviour led to the investigation of statistical models. Predictive statistical models enable the anticipation of certain aspects of human behaviour, such as goals, actions and preferences. In this paper, we motivate the development of these models in the context of the user modeling enterprise. We then review the two main approaches to predictive statistical modeling, content-based and collaborative, and discuss the main techniques used to develop predictive statistical models. We also consider the evaluation requirements of these models in the user modeling context, and propose topics for future research. <s> BIB002 </s> DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Evaluation Techniques <s> The increasing availability of a large number of interactive multi-media information services means that users have a large and diverse collection of choices open to them. This diversity and choice may present navigation difficulties to users which can dissuade them from using such services. One method of assisting users to navigate through large collections is to use information filtering to extract only the information relevant to an end-user according to his/her long-term preferences. In this paper, we describe a mechanism to acquire a user's long-term preferences (user profile), and then show how the acquired profile may be used to recommend selections that may be of interest to the user. The profile is acquired on the basis of a user's habits using a Heuristic-Statistical approach, and is used to create selection indices which are then used during on-line interactions to recommend selections. Our mechanism has been incorporated into an experimental Video On Demand (VOD) service that is implemented using a client-server architecture. The profile acquisition component is incorporated into a VOD server on a multi-tasking machine, while the VOD user interface resides on a personal computer. Our mechanism for acquiring profiles and making recommendations has been quantitatively evaluated on the basis of data collected about movie preferences. <s> BIB003
The evaluation of statistical user models is an ongoing area of research and a variety of techniques can be found in the literature. As described in BIB002 , these are typically methods adapted from other fields of research such as Information Retrieval or Machine Learning. A distinction can be made between direct methods of evaluation using metrics such as Accuracy or Precision and Recall and indirect methods which employ metrics such as Utility. Direct evaluation methods assess the user model by testing the quality of its predictions. Precision and Recall for instance can be used to test a Recommender system by measuring what proportion of recommended items is relevant and what proportion of the relevant items is recommended BIB003 . Similarly, Accuracy measures what percentage of predicted events actually occur. Accuracy is a well-known measure in Spoken Dialogue Systems for measuring the performance of speech recognizers and it is also widely used in User Modelling for measuring the performance of predictive models BIB001 . Indirect evaluation metrics such as Utility attempt to measure the quality of a user model by evaluating the effect of the model on the system performance. The goal is to obtain a measure of the benefit or penalty involved in making a "good" or "bad" prediction. Indirect methods can be said to be closer to studies with real users, because they take the user model's impact on the user into account BIB002 . Simulation-based strategy learning is an interesting application from a statistical user modelling viewpoint, because it has been approached using both direct as well as indirect methods of evaluation. Direct methods have been used to evaluate the quality of the simulated user output, and indirect methods have been used to evaluate the performance of the strategies that we can learn with the available user models. The latter part is particularly interesting, because it is one of the applications of statistical user modelling where the utility of some user model A has been evaluated by testing the system performance on a competing user model B.