To this end, rather than utilizing a pipelined approach, we aim to directly finetune language models in a task-aware manner such that they can maximize a given utility function. We observe that large language models can already be formulated within a Markov decision processes (MDP) as capturing both the dynamics and policy for a decision-making task, where dialogue history serves as state, and the agent's utterances serve as actions.
We could utilize this observation by finetuning the models directly with online RL, but the need for human-in-the-loop training makes this difficult. Offline RL methods [1]}, [2]}, [3]}, [4]} provide an alternative approach, but typically require value function estimation, which is not straightforward to perform with a language model.
Instead, we propose a conditional imitation learning strategy coupled with a novel task relabeling approach that can finetune language models from offline data, such that the model still represents the joint distribution over dialogues, but tilts this distribution toward dialogues with a high reward. This amounts to a task-aware finetuning strategy that integrates task information into the model.
The main contribution of our work is CALM (Context-Aware Language Modeling), a framework for end-to-end goal-directed dialogue generation.
CALM unifies the traditional language modeling objective with task-specific supervision, where a language model is interpreted as a joint representation of dynamics and policies in an MDP, and the finetuning process utilizes a conditional imitation learning objective with a novel task relabeling strategy that teaches the model how to generate high-utility dialogues (see Figures REF  and REF ).
Because CALM interprets the language model as both a dynamics model and a policy, it can be used as either a model-free method, where the dynamics are discarded and the policy component is used to greedily generate responses, or as a model-based method, where the dynamics component can be used to plan at test-time.
We empirically evaluate CALM on AirDialogue [5]}, the largest dataset for goal-oriented dialogue based-on a flight-booking task. CALM improves the task success by 10% over the previous state-of-the-art method [6]} following the evaluation protocol proposed by [5]}, achieving the first-ever human-level performance on this dataset.
<FIGURE>