Submissions to past toxicity detection tasks at SemEval, such as OffensEval and OffensEval-2020, highlight how effective BERT can be for toxicity detection. [1]} used a fine-tuned BERT model to achieve state-of-the-art results on Subtask A at OffensEval, and seven of the top ten teams used BERT. Similarly, the top ten teams on Subtask A at OffensEval-2020 all used BERT, RoBERTa, or XLM-RoBERTa [2]}, sometimes as part of ensemble models with CNNs and LSTMs. [3]} submitted the best performing model, which used an ensemble of ALBERT [4]} models of different sizes. The success of these models in toxicity detection tasks led us to choose to use a BERT-based model for this work.
