Optimizing a dialog system against human users requires many interactions between the system and humans and often incurs real-world costs for failures (Figure REF  (Left)).
User simulators provide an inexpensive alternative to RL-based policy optimization (Figure REF  (Middle)). The user simulators, in theory, do not incur any real-world cost and can provide unlimited simulated experience for RL. But user simulators usually lack the conversational complexity of human interlocutors, and the trained task bot is inevitably affected by biases in the design of the simulator. [1]} demonstrates a significant discrepancy in a simulator-trained task bot when evaluated with simulators and with real users.
<FIGURE>