We train all models with identical hyperparameters.
We perform gradient descent using the AdamW optimizer [1]} with the learning rate set to \(3 * 10^{-5}\)  for BERT and \(10^{-4}\)  for the classifier on top.
We apply gradient clipping of 1.5 and a learning rate warm up of 300 steps using a linear scheduler.
We select negative log likelihood as the loss function.
The tokenizer modules often split single words to multiple subwords. For this task we only need to predict punctuations after words (between white spaces).
We mask the loss function for every other subword.
It is a common practice to intermittently freeze and unfreeze the weights of the transformer model, while training the fine-tuning linear layers situated at the top of the whole architecture.
We found that it is best to have the transformer model unfrozen from the very first epoch and therefore update its parameters along with the linear layers.
We trained the models for 12 epochs with a batch size of 4 and applied early stopping based on the validation set.
We used the validation set to tune the sliding window step size, that is responsible for getting multiple predictions for a single token.
All experiments were performed using a single Nvidia GTX 1070 GPU with one epoch taking 10 minutes.
Our longest training lasted for 2 hours.
