To execute rollout planning, we need a reward predictor which can estimate whether a given dialogue is a successful example of task completion or not. In the case of AirDialogue, we found that the most robust way to estimate this reward is the following: we first fine-tune a RoBERTa-base model (123M parameters) to predict the customer's ground-truth goal and flight requirements from the set of dialogues in the training set. We used the Huggingface Transformers library's implementation of RoBERTa [1]}. We do not filter the training-set when training this model. Once this model is trained, our procedure for predicting dialogue success is the following:
