BERTOverflow
Transformers [1]} are attention-based DL models that can handle sequential data. These trained models are used for various language-specific downstream tasks after fine-tuning the pre-trained models for the specific task-related domain (e.g., sentiment detection, machine translation). Our selection of these five transformer models is based on previous findings in SE classification tasks. BERT has shown massive improvement over other deep models for sentiment analysis tools in SE [2]}[3]}. Other variants of BERT, i.e., RoBERTa, XLNet, etc., are also used in the SE classification task. RoBERTa, the most optimized BERT model, has shown better performance in multiple SE classification tasks. For example, Uddin et al. [4]} found that RoBERTa performed better than other transformer models in six different datasets for sentiment detection. In another recent study, Dai et al. [5]} showed RoBERTa's superiority in aspect level software sentiment analysis. Moreover, we find that other transformer models, such as DistilBERT and XLNet, also perform as well as the BERT model [6]}. Besides this, Tabassum et al. [7]} found that the SO domain specific model BERTOverflow had improved the performance of SO related classification tasks. This motivated us to study these five models for our IoT security aspect classification task. We leave the exploration of recent state-of-the-art models like GPT2 and T5 as our future work. In Table REF , we show the details of the five studied models in this paper.
<FIGURE><TABLE>Fine-Tuned Architecture. Table REF  shows the architectural details of the used Transformer models from the
hugging face
transformer library, an open-source community centered on pretrained
transformer libraries. BERT architecture provides the contextual embedding for
each word in a sentence. For classification purposes, a classifier is used
at the top of the model that takes the embedding of sentences as input and
outputs the predicted weights for each category (Security and Non-Security). Figure REF  (right) shows the used architecture of the classifier. It consists of a dense layer with \(Tanh\)  activation followed by a dropout layer, a dense layer with \(SoftMax\)  activation. Figure REF  (left)
shows how we used the BERT architecture for our security aspect detection.
First, a tokenizer takes the input sentences and represents them as tokenized
sentences. Then, these tokenized sentences are fed into the BERT model. Finally, the
embedded sentences are passed through the classifier to get the final
output (security or non-security).
Following the sequence classifier model from hugging face, we apply \(Tanh\)  activation. In addition, we use the dropout layer to reduce overfitting. To finetune the model on our dataset, we initialize the classifier randomly where the BERT is initialized from pretrained models.
For each deep learning model, following literature in text classifications [8]}, [9]}, [10]},
we tuned the following hyperparameters for optimal
performance: [(1)]

