
We fine-tune BERT as a masked language model on task corpus for 3 epochs without next sentence prediction loss. This provides no noticeable increase in F-score.

Teacher forcing (scheduled sampling) [1]} is a method of improving performance for autoregressive models. As LaserTagger predictions explicitly depend on predictions from the previous step, errors accumulated earlier may lead to a completely wrong result. Teacher forcing aims to close the gap between training and inference by supplying a model with its own (possibly wrong) predictions at a specific rate during training. This allows the model to explore more of its state space during training, which in turn increases its robustness during inference. We linearly decay teacher forcing rate from 1 (only correct labels) to 0 (only model predictions) during training.
The downside of this approach is a significant slowdown in training as we do not utilize the transformer's capability to process the whole sequence at once.

Label smoothing was used to lower the number of false-positive predictions and mitigating minor data discrepancies like inconsistent inclusion of dots and quotations in propaganda spans, which were caused by limiting spans to sentences and other preprocessing imperfections.

