The neural network was implemented in PyTorch [1]}, and we used the tokenizer, pretrained WordPiece, and positional embeddings and pre-trained BERT from the library pytorch-pretrained-berthttps://github.com/huggingface/pytorch-pretrained-BERT.
Following the recommendation for fine-tuning in the original BERT approach [2]}, we trained our classifier with a batch size of 32 for 2 epochs. The dropout probability was set to 0.1 for all layers. Adam optimizer was used with a learning rate of 2e-5. The training was carried out on an Nvidia 1070Ti GPU; only taking about 6 minutes in total.
