It is important to note that the hTER scores used in the initial <cite>HyTER</cite> evaluation were produced by experienced LDC annotators, while the WMT16 Direct Assessment (DA) adequacy judgments were collected from non-experts through crowd-sourcing (Bojar et al., 2016) .