We study the performance accuracy of the studied DL transformer models to detect IoT security-related sentences in SO. Thus, we create a benchmark dataset of 5,919 sentences from our IoT dataset of 53K posts.
First, we use nltk [1]} to segregate sentences from the 53K IoT posts, which
results in 672,678 sentences (i.e., an average of 13 sentences per post). Then we randomly select 5,919 non-overlapping sentences and label those.
This sample size is statistically significant with a 99.99% confidence level and
2.5 confidence interval. The first two authors then manually label each sentence as 1 (i.e., contains security discussion)
or 0 (otherwise). Prior to labeling, the authors consult together to establish a coding guideline. The second author has a PhD and has published peer-reviewed high-impact papers in software security. Following that, each author separately labels all sentences based on the guidelines. The agreement analysis is computed between the authors based on two metrics (Cohen Kappa and Percent Agreement).
The disagreements are resolved by mutual discussion between the authors.
Finally, the random samples have 1049 sentences labeled as security (17.7%). As a post consists of multiple sentences (i.e., security and non-security related sentences) and security-related sentences can be anywhere inside a post, the positive rate (% of security-related samples) is low. One prior study on SO security was conducted by Uddin et al. [2]}, where there were only 4% (197 samples) security-related samples out of 4522 samples. In Table REF , we show summary statistics of the IoT Security dataset. The last two columns of Table REF  show the agreement
level between the two coders during their manual labeling. The Cohen kappa value is 0.94 and the percent agreement is 97.6%. According to Viera at el. [3]}, a kappa value between 0.81 and 0.99 denotes a perfect agreement.
