All models are evaluated using macro \(F_1\) -score (\(F\) ) over the 4 classes. Similarly to Courtland et al., our work is focused on the performance of punctuation marks and as EMPTY labels constitute 85% of all labels, we report the overall \(F_1\) -score without EMPTY.
We evaluated both cased and uncased variations of BERT and generally we have found that the uncased model is better than its cased variant for this task.
This was an expected conclusion, as we lowercased the entire corpus with the purpose of eliminating bias around the prediction of periods.
For all setups, we selected the best performing models on the validation set by loss and by macro \(F_1\) -score and evaluated them independently on the test set. On the Ted Talks dataset, our best performing model was an uncased variation of BERT that achieved on par performance with the current state-of-the-art model [1]}, having a slightly worse macro \(F_1\) -score of 79.8 (0.8 absolute and 0.9975% relative difference) with 10 epochs of training and 64 predictions/token. All results on the Ted Talks dataset are summarized in Table REF .
<TABLE><FIGURE>