Those that conduct manual assessment of the content, typically use a single reference summary, either directly (Celikyilmaz et al., 2018; Tan et al., 2017) or through questions <cite>(Narayan et al., 2018b</cite>,c) and thus are also likely to exhibit reference bias. In this paper we propose a novel approach for manual evaluation, HIGHlight-based Referenceless Evaluation of document Summarization (HIGHRES), in which a summary is assessed against the source document via manually highlighted salient content in the latter (see Figure 1 for an example).