CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. 371 Application of International Patent Application No. PCT/JP2019/005375, filed on 14 Feb. 2019, which application claims priority to and the benefit of JP Application No. 2018-032554, filed on 26 Feb. 2018, the disclosures of which are hereby incorporated herein by reference in their entireties.

TECHNICAL FIELD

The present invention relates to a summary evaluation apparatus, a method, a program, and a storage medium, and in particular relates to a summary evaluation apparatus, a method, a program, and a storage medium for evaluating a system summary.

BACKGROUND ART

Conventionally, in the field of natural language processing in which language is automatically processed using a computer, there has been a technique for automatically grading a system summary using a correct summary created by a human (hereinafter referred to as a reference summary) when the reference summary is provided.

As a method for automatically evaluating a summary, a method is often used in which a determination is made based on the degree of matching between a system summary and a reference summary for smaller fragments in a sentence, such as n-grams or tuples, as in ROUGE (NPL 1) or Basic Elements.

CITATION LIST

Non Patent Literature

[NPL 1] Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Proc. of Workshop on Text Summarization Branches Out. pages 7481.

SUMMARY OF THE INVENTION

Technical Problem

Evaluation methods such as ROUGE and Basic Element are used to evaluate system summaries according to whether or not there are fragments smaller than sentences in the text. On the other hand, many summarization systems generate summaries by extracting meaningful fragments in the text, such as “sentences” or “phrases”.

Accordingly, in consideration of performing system error analysis, analysis can be performed more intuitively by seeing whether or not extraction of “sentences” or “phrases” was successful than by seeing whether or not extraction of n-grams or tuples was successful. However, in automatic summary evaluation, ROUGE and Basic Elements are de-facto standards, and thus it is required that their scores are improved.

The present invention was achieved in order to resolve the foregoing problems, and it is an object thereof to provide a summary evaluation apparatus, a method, a program, and a storage medium, according to which it is possible to accurately evaluate a system summary with consideration given to the meaning of each unit.

Means for Solving the Problem

In order to achieve the above-described object, a summary evaluation apparatus according to a first aspect of the invention is constituted by including: a unit division portion configured to divide a document to be summarized and a system summary generated in advance for the document to be summarized, into predetermined units; an oracle generation portion configured to, for each reference summary created in advance for the document to be summarized, generate an oracle based on the reference summary and a set of the units of the document to be summarized, the oracle being a subset of units of the document to be summarized that satisfy a length restriction and maximize a score of an evaluation function for the subset of units with respect to the reference summary; an oracle unit score determination portion configured to determine a score of each unit included in a set of the oracles based on the oracles generated for the reference summaries; a system unit score determination portion configured to, for each unit of the system summary, obtain a corresponding unit from among the units included in the set of oracles and determine a score of the unit of the system summary; and an evaluation score determination portion configured to determine a score of the system summary based on the scores of the units of the system summary.

A summary evaluation method according to a second aspect of the invention includes and executes the steps of: a unit division portion dividing a document to be summarized and a system summary generated in advance for the document to be summarized, into predetermined units; an oracle generation portion generating, for each reference summary created in advance for the document to be summarized, an oracle based on the reference summary and a set of the units of the document to be summarized, the oracle being a subset of units of the document to be summarized that satisfy a length restriction and maximize a score of an evaluation function for the subset of units with respect to the reference summary; an oracle unit score determination portion determining a score of each unit included in a set of the oracles based on the oracles generated for the reference summaries; a system unit score determination portion obtaining, for each unit of the system summary, a corresponding unit from among the units included in the set of the oracles, and determining a score of the unit of the system summary; and an evaluation score determination portion determining a score of the system summary based on the scores of the units of the system summary.

A program according to a third aspect of the invention is a program for causing a computer to function as the portions of the summary evaluation apparatus according to the first aspect of the invention.

A storage medium according to a fourth aspect of the invention is a storage medium storing a program for causing a computer to function as the portions of the summary evaluation apparatus according to the first aspect of the invention.

Effects of the Invention

According to the summary evaluation apparatus, method, program, and storage medium of the present invention, the following effect can be obtained. A system summary can be accurately evaluated with consideration given to the meaning of each unit by dividing a document to be summarized and a system summary generated in advance for the document to be summarized, into predetermined units, generating an oracle for each reference summary created in advance for the document to be summarized, based on the reference summary and a set of the units of the document to be summarized, the oracle being a subset of units of the document to be summarized that satisfy a length restriction and maximize a score of an evaluation function for the subset of units with respect to the reference summary, determining the scores of the units included in the set of oracles based on the generated oracles, obtaining, for each unit of the system summary, a corresponding unit from among the units included in the set of oracles and determining the score of the unit of the system summary, and determining the score of the system summary based on the scores of the units of the system summary.

