Methods based on similar principles have previously been proposed in the deep RL community for simple parametric tasks, such as goal-reaching or linearly-parameterized reward functions [1]}, [2]}, [3]}. However, the dialogue task relabeling that we employ is particularly effective in our setting, since there may be exponentially many contexts that are optimal for a given dialogue (e.g., many different flight tables for a flight booking task), in contrast to the simpler task parameterizations used in prior work, where for example only one goal might be optimal for a given trajectory (the one that is reached). As a result, this technique not only allows us to turn sub-optimal task data into optimal data, but it also allows us to greatly increase the number of optimal task examples from which we can learn, which we will show leads to a large performance improvement.
