In order to quantitatively evaluate the consistency of the labelling of the three users, additional reliability measures were obtained.
In particular, the inter-rater and intra-rater agreement has been computed using the linearly weighted kappa coefficient [1]}.
The intra-rater measures have been obtained using a set of 200 scenarios labelled twice by each subject. The inter-rater agreement has been computed from 522 common scenarios labelled by the three subjects.
The results of these measures can be observed in Table REF .
The diagonal cells of the Table REF  represent the intra-rater reliability, while the remaining cells show the inter-rater agreement for every pair of subjects.
As can be seen, the inter-rater consistency ranges from \(0.61\)  to \(0.71\) , showing substantial agreement among the three users according to [2]}.
In addition, intra-rater reliability for all the subjects are above \(0.75\) , which indicates substantial to almost perfect agreement, i.e. high reliability.
<TABLE>