The interannotator agreement results, using Fleiss’s [1]} kappa and prevalence- and bias-adjusted Fleiss’s kappa [2]} for multiple annotators, are displayed in Figure REF .
Fleiss’s kappa is generally viewed as the gold standard measure for investigating agreement across many annotators, and it represents the degree of agreement beyond what is expected by chance. This measure though, is heavily influenced by the prevalence of positive cases. Given the subjective nature of our task, and the fact that positive cases are not prevalent given the often rarity of moral rhetoric [3]}, we use PABAK which adjusts kappa for prevalence and bias. As expected, given the low base rate of moral language across the various subrredits (i.e. low positive cases), most reported kappa's are low. However, once adjusted for the issue of prevalence, we see medium to high agreements across the subreddits.
<FIGURE>