Finally, in order to facilitate research into annotator response patterns and bias, as recommended by [1]}, we provide psychological and demographic metadata of our annotators. The background and biases of human annotators have been shown to impact their annotations [2]}, [3]}, [4]}, [5]} with particularly damaging effects that amplify pre-existing biases [6]}, [7]}.
Annotators' biases may be particularly relevant in domains characterized by high subjectivity, such as moral values [8]}, [9]}, [10]}. While, for example, an annotator’s political ideology might not have a substantial influence on how they annotate “positive” and “negative” sentiment in a corpus of hotel reviews, it seems likely that their ideology could substantially influence how they annotate expressions of justice and respect in a politically relevant corpora.
