DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Bayesian Networks <s> Speech enabled interfaces and spoken dialog systems are mostly based on statistical speech and language processing modules. Their behavior is therefore not deterministic and hardly predictable. This makes the simulation and the optimization of such systems performances difficult, as well as the reuse of previous work to build new systems. In the aim of a partially automated optimization of such systems, this paper presents a formalism attempt for the description of man-machine spoken communication in the framework of spoken dialog systems. This formalization is partly based on a probabilistic description of the information processing occurring in each module composing a spoken dialog system but also on a stochastic user modeling. Eventually, some possible applications of this theoretic framework are proposed <s> BIB001 </s> DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Bayesian Networks <s> Speech enabled interfaces are nowadays becoming ubiquitous. The most advanced ones rely on probabilistic pattern matching systems and especially on automatic speech recognition systems. Because of their statistical nature, performances of such systems never reach one hundred percent of correct recognition results. Performances are linked to environmental noise and to intraand inter-speaker variability of course, but also to the acoustical similarities inside the vocabulary of allowed speech entries, which is usually contextual in the case of man-machine dialogue systems. A good dialogue strategy should therefore dynamically handle the potentiality of recognition errors. In this paper, we compare different methods to model ASR systems in the framework of automatic dialogue strategy optimization and we especially emphasize on a contextdependent ASR modeling method. <s> BIB002 </s> DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies <s> Bayesian Networks <s> The design of Spoken Dialog Systems cannot be considered as the simple combination of speech processing technologies. Indeed, speech-based interface design has been an expert job for a long time. It necessitates good skills in speech technologies and low-level programming. Moreover, rapid development and reusability of previously designed systems remains uneasy. This makes optimality and objective evaluation of design very difficult. The design process is therefore a cyclic process composed of prototype releases, user satisfaction surveys, bug reports and refinements. It is well known that human intervention for testing is time-consuming and above all very expensive. This is one of the reasons for the recent interest in dialog simulation for evaluation as well as for design automation and optimization. In this paper we expose a probabilistic framework for a realistic simulation of spoken dialogs in which the major components of a dialog system are modeled and parameterized thanks to independent data or expert knowledge. Especially, an Automatic Speech Recognition (ASR) system model and a User Model (UM) have been developed. The ASR model, based on articulatory similarities in language models, provides task-adaptive performance prediction and Confidence Level (CL) distribution estimation. The user model relies on the Bayesian Networks (BN) paradigm and is used both for user behavior modeling and Natural Language Understanding (NLU) modeling. The complete simulation framework has been used to train a reinforcement-learning agent on two different tasks. These experiments helped to point out several potentially problematic dialog scenarios. <s> BIB003
Pietquin  BIB001 BIB002 BIB003 combines features from Scheffler and Young's work with the Levin Model in an attempt to avoid the manual effort involved in building networks of choice points while ensuring that actions are consistent with the user's goal. The core idea of his work is to condition the set of probabilities selected by Levin et al. on an explicit representation of the user goal and memory. As in Scheffler and Young's work, the user goal is simply a table of <slot name, slot value> pairs with associated status variables. Pietquin uses the status variables to rank the user's priority for each slot value and to track how often a certain piece of information has been mentioned by the user during the conversation with the system. Although these models of the user goal and memory are rather crude, they do acknowledge that a better understanding of the user state and its effect on conversational behaviour is needed to make accurate predictions of user behaviour. In his work, Pietquin hand-selects all model parameters using empirical principles and common sense. Suitable probability values (eg. Equation 17) are set by hand. Pietquin suggests that in future work the conditional dependencies in the user model be graphically visualised and implemented as a Bayesian network, as illustrated in Figure 7 . According to Pietquin, the network can be easily extended to include factors such as the level of cooperativeness, the degree of initiative, etc. It is however not intuitively clear which parameters are potential candidates for modelling these user attributes and how they can be trained on human-computer dialogue data. 