We emulated the expert annotation process carried out by <cite>Wachsmuth et al. (2017a)</cite> on CrowdFlower in order to evaluate whether lay annotators suffice for a theory-based quality assessment.