
where \(\pi _{\theta }(a_t|o_t)\)  represents a policy that generates new dialogue based on the observed context and dialogue history, and \(\mathcal {T}_{\theta }(\tau _{<t+1}|\tau _{<t}, a_t)\)  represents the observed dynamics characterizing human responses, and \(\theta \)  denotes parameters in \(\pi \)  and \(\mathcal {T}\) . Note that \(\tau _{<t}\)  consists only of the conversation history, and does not contain any task-specific context.
A naïve approach to train dialogue systems is to jointly parameterize both \(\pi \)  and \(\mathcal {T}\)  as one language model, and optimize Equation REF  on pre-collected conversations \(\mathcal {D}^\mathrm {off}\) . This method corresponds to behavioral cloning (BC) [1]}.
