The first category uses proxy evaluation tasks, such as question answering (QA), and a masked token prediction cloze task.
Given \((\mathcal {D}, Y)\)  where \(Y\)  is a model-generated summary of \(\mathcal {D}\) , the QA-based metrics assign a quality score to \(Y\)  by measuring how accurate, in terms of precision/recall/F-score, a QA system can answer questions generated from \(\mathcal {D}\)  based on \(Y\) , or vice versa. For example, the SummaQA metric [1]} answers questions derived from \(\mathcal {D}\)  based on \(Y\) , whereas the FEQA metric [2]} answers questions from \(Y\)  based on \(\mathcal {D}\) .
The cloze-based metrics assign the quality of a summarizer by measuring the performance boost gained by a pre-trained language model with access to its generated summaries while performing the token prediction task on the source documents [3]}. As illustrated by the two examples in Figure REF , the BLANC-help metric (Left) is defined by the difference in accuracy of two reconstructions of masked tokens: with summary versus filler concatenated in front of the sentence with masked tokens, where the model input is a summary (or filler) and sentence with masked (grey) tokens, and the output is the unmasked tokens. The BLANC-tune metric (Right) is defined by the difference in accuracy of two reconstructions of masked tokens: with model tuned on the summary versus with the original model, where both models are given the same input consisting of a sentence and its masked (grey) tokens, and each model outputs the unmasked tokens.
<FIGURE>