Our goal is to enable end-to-end training of goal-directed dialogue agents. In these settings, an agent aims to complete a particular task with its utterances [1]}. Goal-directed agents have been explored in contexts such as personal assistants [2]}, [3]}, [4]}, recommendation systems [5]}, [6]}, education [7]}, and negotiation [8]}, [9]}.
While there are multiple approaches to constructing dialogue agents, in this work we frame the problem of generating dialogue as a sequential decision making problem within a (partially observed) Markov Decision Process (MDP) [10]}, [11]}. Prior works that utilize such an MDP formulation typically aim to train a dialogue management system [12]}, in which the agent reasons about higher-level abstractions of the state of the conversation, and language generation is performed using a downstream procedure. Dialogue management systems have been trained using techniques such as online reinforcement learning via policy gradients [13]}, [8]}, off-policy reinforcement learning [15]}, [16]} or actor-critic methods [17]}. Our method differs from dialogue management systems in that CALM is an end-to-end system optimized for successful task completion, and performs both high-level decision making and language generation.
