All five transformer models show better performances in terms of all metrics than the baseline models from Uddin [1]} (SVM and Logit). The last two rows in Table REF  show the performance change (in percentages) we observed in the best performing transformer model (RoBERTa) compared to the two baselines. For example, the best
performing transformer model is RoBERTa with an F1-score of 0.935, which outperforms the baselines SVM by 8.0%, and Logits by 8.2%. Higher performances mean that transformers are precisely (i.e., around 6% higher precision) discovering (i.e., around 10% higher recall) security-specific sentences. It is because these models can understand the contextual meaning in a sentence. For example, the following non-security related sentence “My goal is to create a sort of password-like system where if you push the right buttons in the right order then a button will light up.” The comments describe the procedure of a system using the references `password like' that have no relation with security. The baselines model erroneously labeled it as 0 due to the misleading keyword- `password like' where the transformer models discerned the context. Hence, it found no security threats in this sentence. This superiority of these models over baselines is not surprising given that the transformer models are generally found to have offered better performance than state-of-the-art classifiers for multiple other text classification tasks in software engineering (e.g., sentiment detection [2]}).
