The raters evaluated a total of 250 questions generated for 25 different claims.
As they assigned Boolean values to intelligibility (I), clarity (C), and relevance (R), similar distributions with minor disagreements led to low Fleiss-\(\kappa \)  and Krippendorff-\(\alpha \)  scores.
The Fleiss-\(\kappa \)  scores are 0.48 for intelligibility, 0.43 for clarity, 0.32 for relevance and  0.26 for informativeness. However, as reported in previous work, chance-adjusted scores can be low despite a high agreement due to their inappropriateness when assessing variables with imbalanced marginal distributions [1]}, [2]}, [3]}, [4]}. Therefore, we have computed the free marginal multi-rater kappa scores [1]}, which are equal to 0.88 for intelligibility, 0.74 for clarity, 0.58 for relevance and 0.32 for informativeness.
