We studied four optimized transformers- i.e., BERT[1]}, RoBERTa[2]}, XLNet[3]}, and DistilBERT[4]} for this research question. Transformer-based advanced language models [5]} are found to outperform other models in text classification (e.g., sentiment detection in SE [6]}). In addition, recent studies suggest that a domain-specific DL model is more effective than a general-purpose deep learning model [7]}. Thus, for this research question, we also employ the BERTOverflow model, which has been pre-trained on SO dumps [8]}. Indeed, in our security aspect detection, we find that the transformer models outperform the baseline SVM, which was found to be the most effective in security aspect detection by Uddin and Khomh [9]} in a general purpose API review dataset. RoBERTa shows the best performance among the four transformer models, with an F1-Score of 0.935. We name this trained model SecBot.
