We also repeated the annotator ensemble experiment of<cite> Wulczyn et al. (2017)</cite> on 8K randomly chosen comments of W-ATT-TEST (4K comments from random users, 4K comments from banned users). 19 The decisions of 10 randomly chosen annotators (possibly different per comment) were used to construct the gold label of each comment. The gold labels were then compared to the decisions of the systems and the decisions of an ensemble of k other annotators, k ranging from 1 to 10.