We build a machine-learned agent based on pre-trained BERT [1]}.We use the HuggingFace implementation of BERT-base-cased model [2]}: hidden size 768, intermediate size 3,072, max position embeddings 512, attention heads 12, and hidden layers 12. We consider 10 previous actions in the dialog history. We train our model on 4 Nvidia P100 GPUs. Batch size is 16 (4 per GPU). We train the model for 20 epochs and select the model checkpoint that has the highest click level accuracy on our Dev set. The training process takes about 2 hours to finish. Action prediction predicts the API or template to invoke, given the previous user and agent context (including information retrieved from previous API calls). Query prediction locates the query for API calls, such as “Starbucks" in the user utterance “I want to go to Starbucks.” Parameter prediction assigns appropriate variables retrieved by API calls to actions (APIs and templates), for example, the machine selects the API return value “13 min" to fill in the template It is {} away.
