Prior works primarily evaluate bots for the flight agent through “self-play" [1]}, [2]}. We follow the same evaluation protocol in our work. Basically, we train a bot to play the role of the customer during evaluation and compute task success by simulating conversations against this bot. We run all self-play evaluations on the same subset of 1,000 dialogue scenarios, randomly selected from the validation set.
