Our customer bot is fine-tuned from GPT2-small (124M parameters), using the standard language modeling objective. We used the Huggingface Transformers library's implementation of GPT2 [1]}. The customer's flight requirements are provided to the model as a prefix to the dialogue, which formatted as a comma separated list consisting of the customer's goal and flight requirements.
We trained the customer bot for maximum 10 epochs with early stopping on the filtered dataset. For training, it takes around 1 day on 4 GPUs. Specifically, we trained using Adam with learning rate 1e-4 and batch size 8.
Our customer bot achieves a perplexity of 1.47 on the development set and a BLEU score of 38.5.
