All our flight agent bots are fine-tuned from GPT2-small (124M parameters) using the standard language modeling objective. We used the Huggingface Transformers library's implementation of GPT2 [1]}. Similar as the customer bot, we trained for maximum 10 epochs with early stopping on the filtered dataset, which takes roughly 1 day on 4 GPUs. Specifically, we trained using Adam with learning rate 1e-4 and batch size 8. We implement the final action prediction as a sequence of tokens generated at the end of each dialogue. The flight table is passed to the model as a prefix of flight embeddings, where each embedding is produced by summing embeddings corresponding to each attribute of a given flight (e.g., flight arrival/departure day/location, flight price, etc.).
