In this section, we present our method for goal-oriented dialogue systems, Context-Aware Language Modeling (CALM).
CALM interprets a language model as a combination of a policy and a dynamics model in the POMDP formulation of a dialogue task, as described in Section . Under this interpretation, naïve supervised finetuning on the dialogue dataset can be viewed as behavioral cloning (BC) [1]}. However, BC only imitates data and does not necessarily produce a good policy in terms of completing tasks. We propose to improve the policy by utilizing a task relabeling strategy (described in Section REF ), analogous to prior task relabeling approaches [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}.
This relabeling procedure augments the data with examples of near-optimal utterances, making the language model more task-aware.
However, we find several shortcomings with this approach alone and propose the following improvements. First, an expressive language model is liable to ignore the task context, which we address by proposing an auxiliary loss (Section REF ) that forces the model to utilize this information. Second, learning from structured task information is difficult and can result in models that fail to capture complex task structure, so we propose a task pre-training procedure to improve the learnability (Section REF ). Finally, to further improve performance we use a model-based planning procedure (Section REF ) on top of the proposed method that samples multiple dialogues in parallel and selects the most promising candidates.
