We first use the newly created IoT Security dataset
from Section REF  to get the best performing transformer model. For experimentation, similar to Uddin and Khomh [1]}, we use a stratified k-fold cross-validation ( k = 10). Each fold divides dataset \(D\)  into \(k\)  disjoint folds, each of
approximately equal size.
\(D_i\cap D_j=\phi ~~~\forall ~i,j\in k~and ~i\ne j \)  and
\(D_1\cup D_2\cup ...\cup D_k = D\) . To avoid overfitting: we train the models \(k\)  times each time we hold one of the \(k\)  folds for validation, and we use the rest \(k-1\)  folds for training. The value of \(k\)  is a hyperparameter. In our experiments, we set K as 10. In Figure REF , we show 10-Fold cross validation techniques. We run the model 10 times. In each iteration, we train the model on nine different folds and test the trained model on the last fold. Then, we calculate the average performance of these 10 iterations. For example, according to the Figure REF , at the first iteration, we pick `Fold-2', `Fold-3', `Fold-4', `Fold-5', `Fold-6', `Fold-7', `Fold-8', `Fold-9', and `Fold-10' to train the model, and we test this model on the `Fold-1'. Therefore, in our case, we fine tune our model during each training (i.e., k times) and then test the tuned model on the test fold.
To tune the hyperparameter, we follow grid search [2]} tuning techniques. For example, we set learning rate as 1e-5, 2e-5, and 3e-5 and select the one that has highest performance for each model. For the two baselines (Logit and SVM), we hyper tuned all the parameters used by Uddin and Khomh [1]}, [4]}. We find the performance of transformer models on benchmark dataset and select the best performing model based on the F1-Score. Then, we test the trained model on our Judgmental dataset to check either the model learn to generalize the security discussion in IoT domain.
<TABLE>