Automatic evaluation of summarizers, especially the abstractive summarizers, is challenging [1]}. A common approach is to collect for each test case multiple human generated summaries as references, then measure the \(N\) -gram overlap between the machine-generated summaries and the references using metrics such as ROUGE [2]}.
However, this approach fails to evaluate the factuality of machine-generated summaries. According to [1]}, 30% of summaries generated by abstractive summarizers contain factual inconsistencies, caused by distorted or fabricated facts about the source documents, as illustrated by the example in Figure REF .
<FIGURE>