Pretrained transformer models have already shown dominating performances over shallow and deep models in software engineering [1]}. Moreover, recent studies on software engineering have found the effectiveness of domain-specific embedding (e.g., BERT, RoBERTa, XLNet, etc.) than a generic one. For example, Tabassum et al. [2]} shows that the BERTOverflow model that is pre-trained on Stack Overflow dumps beats the performance of efficient transformers like RoBERTa on developer discussions such as Github and Stack Overflow. Therefore, we investigate the possibility of improving the baseline performance to detect security aspects in IoT discussions using studied Transformer Models, e.g., BERT.
