DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Introduction and Overview <s> From the Publisher: ::: This comprehensive state-of-the-art book is the first devoted to the important and timely issue of evaluating NLP systems. It addresses the whole area of NLP system evaluation, including aims and scope, problems and methodology. The authors provide a wide-ranging and careful analysis of evaluation concepts, reinforced with extensive illustrations; they relate systems to their environments and develop a framework for proper evaluation. The discussion of principles is completed by a detailed review of practice and strategies in the field, covering both systems for specific tasks, like translation, and core language processors. The methodology lessons drawn from the analysis and review are applied in a series of example cases. The book also refers NLP system evaluation to the neighbouring areas of information and speech processing, and addresses issues of tool and data provision for evaluation. A comprehensive bibliography and subject index are included as well as a term glossary. This monograph will be a valuable source of inspiration in research, practice, and teaching. <s> BIB001 </s> DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Introduction and Overview <s> We propose a quantitative model for dialog systems that can be used for learning the dialog strategy. We claim that the problem of dialog design can be formalized as an optimization problem with an objective function reflecting different dialog dimensions relevant for a given application. We also show that any dialog system can be formally described as a sequential decision process in terms of its state space, action set, and strategy. With additional assumptions about the state transition probabilities and cost assignment, a dialog system can be mapped to a stochastic model known as Markov decision process (MDP). A variety of data driven algorithms for finding the optimal strategy (i.e., the one that optimizes the criterion) is available within the MDP framework, based on reinforcement learning. For an effective use of the available training data we propose a combination of supervised and reinforcement learning: the supervised learning is used to estimate a model of the user, i.e., the MDP parameters that quantify the user's behavior. Then a reinforcement learning algorithm is used to estimate the optimal strategy while the system interacts with the simulated user. This approach is tested for learning the strategy in an air travel information system (ATIS) task. The experimental results we present in this paper show that it is indeed possible to find a simple criterion, a state space representation, and a simulated user parameterization in order to automatically learn a relatively complex dialog behavior, similar to one that was heuristically designed by several research groups. <s> BIB002 </s> DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Introduction and Overview <s> The fields of user modeling and natural language processing have been closely linked since the early days of user modeling. Natural language systems consult user models in order to improve their understanding of users' requirements and to generate appropriate and relevant responses. At the same time, the information natural language systems obtain from their users is expected to increase the accuracy of their user models. In this paper, we review natural language systems for generation, understanding and dialogue, focusing on the requirements and limitations these systems and user models place on each other. We then propose avenues for future research. <s> BIB003
Research on user modelling has a long history in the field of Spoken Dialogue Systems (SDS) and Natural Language Processing (NLP). Traditionally, the most commonly cited motivation for this work has been to enable dialogue systems to adapt to individual user needs and preferences by consulting a model of their human dialogue partner, similar to the way humans intuitively use mental models of their interlocutor when engaging in a conversation. The majority of research on user modelling in SDS and NLP has always had a strong linguistically motivated focus on non-statistical methods and the modelling techniques employed have often been highly complex in nature and frequently based on manually constructed rules or various forms of logic BIB003 ). In recent years, however, the application of machine-learning approaches to dialogue system design has created a need for simple statistical user models which are probabilistic in nature and trainable on existing human-computer dialogue data. The purpose of user modelling in this new area of SDS research is not to model the precise characteristics of an individual user during the live operation of a system, but to provide the basis for user simulation tools which can be used during the development phase of a new dialogue system BIB002  . Current work on predictive, statistical user modelling in SDS is of particular importance to the design of dialogue management strategies. The task of the Dialogue Manager (DM) in a spoken dialogue system is to control the flow of the dialogue and to decide which actions to take in response to user input. In order to complete these tasks, the DM needs a dialogue strategy that defines when to take the initiative in a dialogue, when to confirm receipt of a piece of information, how to identify and recover from recognition and understanding errors, and so forth. The manual development of a good dialogue strategy is a major design challenge and has been described as "an art, rather than a science" BIB001 for many years. Machine-learning approaches to dialogue management attempt to learn optimal strategies from corpora of real human-computer dialogue data using automated "trial-and-error" methods instead of relying on empirical design principles. However, the size of currently available annotated dialogue corpora is far too small to sufficiently explore the vast space of possible dialogue states and strategies. In addition, no guarantee can be given that the truly optimal strategy is indeed present in a given training corpus and it may thus be argued that an optimal strategy cannot be learned from a fixed corpus, regardless of its size. An interesting solution to this problem is to train a statistical, predictive user model for simulating user responses which can then be used to learn dialogue strategies through trial-and-error interaction between the dialogue manager and the simulated user BIB002     .