Q: Which evaluation methods are used?
Text: Introduction
Summarization of large texts is still an open problem in language processing. People nowadays have lesser time and patience to go through large pieces of text which make automatic summarization important. Automatic summarization has significant applications in summarizing large texts like stories, journal papers, news articles and even larger texts like books.
Existing methods for summarization can be broadly categorized into two categories Extractive and Abstractive. Extractive methods picks up words and sometimes directly sentences from the text. These methods are inherently limited in the sense that they can never generate human level summaries for large and complicated documents which require rephrasing sentences and incorporating information from full text to generate summaries. Most of the work done on summarization in past has been extractive.
On the other hand most Abstractive methods take advantages of the recent developments in deep learning. Specifically the recent success of the sequence to sequence learning models where recurrent networks read the text, encodes it and then generate target text. Though these methods have recently shown to be competitive with the extractive methods they are still far away from reaching human level quality in summary generation.
The work on summarization using AMR was started by BIBREF0 . Abstract Meaning Representation (AMR) was as introduced by BIBREF1 . AMR focuses on capturing the meaning of the text, by giving a specific meaning representation to the text. AMR tries to capture the "who is doing what to whom" in a sentence. The formalism aims to give same representation to sentences which have the same underlying meaning. For example "He likes apple" and "Apples are liked by him" should be assigned the same AMR.
BIBREF0 's approach aimed to produce a summary for a story by extracting a summary subgraph from the story graph and finally generate a summary from this extracted graph. But, because of the unavailability of AMR to text generator at that time their work was limited till extracting the summary graph. This method extracts a single summary graph from the story graph. Extracting a single summary graph assumes that all of the important information from the graph can be extracted from a single subgraph. But, it can be difficult in cases where the information is spread out in the graph. Thus, the method compromises between size of the summary sub-graph and the amount of information it can extract. This can be easily solved if instead of a single sub-graph, we extract multiple subgraphs each focusing on information in a different part of the story.
We propose a two step process for extracting multiple summary graphs. First step is to select few sentences from the story. We use the idea that there are only few sentences that are important from the point of view of summary, i.e. most of the information contained in the summary is present in very few sentences and they can be used to generate the summary. Second step is to extract important information from the selected sentences by extracting a sub-graph from the selected sentences.
Our main contributions in this work are three folds,
The rest of the paper is organized as follows. Section SECREF2 contains introduction to AMR, section SECREF3 and SECREF4 contains the datasets and the algorithm used for summary generation respectively. Section SECREF5 has a detailed step-by-step evaluation of the pipeline and in section SECREF6 we discuss the problems with the current dataset and evaluation metric.
Background: AMR Parsing and Generation
AMR was introduced by BIBREF1 with the aim to induce work on statistical Natural Language Understanding and Generation. AMR represents meaning using graphs. AMR graphs are rooted, directed, edge and vertex labeled graphs. Figure FIGREF4 shows the graphical representation of the AMR graph of the sentence "I looked carefully all around me" generated by JAMR parser ( BIBREF2 ). The graphical representation was produced using AMRICA BIBREF3 . The nodes in the AMR are labeled with concepts as in Figure FIGREF4 around represents a concept. Edges contains the information regarding the relations between the concepts. In Figure FIGREF4 direction is the relation between the concepts look-01 and around. AMR relies on Propbank for semantic relations (edge labels). Concepts can also be of the form run-01 where the index 01 represents the first sense of the word run. Further details about the AMR can be found in the AMR guidelines BIBREF4 .
A lot of work has been done on parsing sentences to their AMR graphs. There are three main approaches to parsing. There is alignment based parsing BIBREF2 (JAMR-Parser), BIBREF5 which uses graph based algorithms for concept and relation identification. Second, grammar based parsers like BIBREF6 (CAMR) generate output by performing shift reduce transformations on output of a dependency parser. Neural parsing BIBREF7 , BIBREF8 is based on using seq2seq models for parsing, the main problem for neural methods is the absence of a huge corpus of human generated AMRs. BIBREF8 reduced the vocabulary size to tackle this problem while BIBREF7 used larger external corpus of external sentences.
Recently, some work has been done on producing meaningful sentences form AMR graphs. BIBREF2 used a number of tree to string conversion rules for generating sentences. BIBREF9 reformed the problem as a traveling salesman problem. BIBREF7 used seq2seq learning methods.
Datasets
We used two datasets for the task - AMR Bank BIBREF10 and CNN-Dailymail ( BIBREF11 BIBREF12 ). We use the proxy report section of the AMR Bank, as it is the only one that is relevant for the task because it contains the gold-standard (human generated) AMR graphs for news articles, and the summaries. In the training set the stories and summaries contain 17.5 sentences and 1.5 sentences on an average respectively. The training and test sets contain 298 and 33 summary document pairs respectively.
CNN-Dailymail corpus is better suited for summarization as the average summary size is around 3 or 4 sentences. This dataset has around 300k document summary pairs with stories having 39 sentences on average. The dataset comes in 2 versions, one is the anonymized version, which has been preprocessed to replace named entities, e.g., The Times of India, with a unique identifier for example @entity1. Second is the non-anonymized which has the original text. We use the non-anonymized version of the dataset as it is more suitable for AMR parsing as most of the parsers have been trained on non-anonymized text. The dataset does not have gold-standard AMR graphs. We use automatic parsers to get the AMR graphs but they are not gold-standard and will effect the quality of final summary. To get an idea of the error introduced by using automatic parsers, we compare the results after using gold-standard and automatically generated AMR graphs on the gold-standard dataset.
Pipeline for Summary Generation
The pipeline consists of three steps, first convert all the given story sentences to there AMR graphs followed by extracting summary graphs from the story sentence graphs and finally generating sentences from these extracted summary graphs. In the following subsections we explain each of the methods in greater detail.
Step 1: Story to AMR
As the first step we convert the story sentences to their Abstract Meaning Representations. We use JAMR-Parser version 2 BIBREF2 as it’s openly available and has a performance close to the state of the art parsers for parsing the CNN-Dailymail corpus. For the AMR-bank we have the gold-standard AMR parses but we still parse the input stories with JAMR-Parser to study the effect of using graphs produced by JAMR-Parser instead of the gold-standard AMR graphs.
Step 2: Story AMR to Summary AMR
After parsing (Step 1) we have the AMR graphs for the story sentences. In this step we extract the AMR graphs of the summary sentences using story sentence AMRs. We divide this task in two parts. First is finding the important sentences from the story and then extracting the key information from those sentences using their AMR graphs.
Our algorithm is based on the idea that only few sentences are important from the point of view of summary i.e. there are only a few sentences which contain most of the important information and from these sentences we can generate the summary.
Hypothesis: Most of the information corresponding to a summary sentence can be found in only one sentence from the story.
To test this hypothesis, for each summary sentence we find the sentence from the story that contains maximum information of this summary sentence. We use ROGUE-1 BIBREF13 Recall scores (measures the ratio of number of words in the target summary that are contained in the predicted summary to the total number of words in the target summary) as the metric for the information contained in the story sentence. We consider the story sentence as the predicted summary and the summary sentence as the target summary. The results that we obtained for 5000 randomly chosen document summary pairs from the CNN-Dailymail corpus are given in figure FIGREF8 . The average recall score that we obtained is 79%. The score will be perfectly 1 when the summary sentence is directly picked up from a story sentence. Upon manual inspection of the summary sentence and the corresponding best sentence from the story we realized, when this score is more than 0.5 or 0.6, almost always the information in the summary sentence is contained in this chosen story sentence. The score for in these cases is not perfectly 1 because of stop words and different verb forms used in story and summary sentence. Around 80% of summary sentences have score above 0.5. So, our hypothesis seems to be correct for most of the summary sentences. This also suggests the highly extractive nature of the summary in the corpus.
Now the task in hand is to select few important sentences. Methods that use sentence extraction for summary generation can be used for the task. It is very common in summarization tasks specifically in news articles that a lot of information is contained in the initial few sentences. Choosing initial few sentences as the summary produces very strong baselines which the state-of-the-art methods beat only marginally. Ex. On the CNN-Dailymail corpus the state-of-the-art extractive method beats initial 3 sentences only by 0.4% as reported by BIBREF14 .
Using this idea of picking important sentences from the beginning, we propose two methods, first is to simply pick initial few sentences, we call this first-n method where n stands for the number of sentences. We pick initial 3 sentences for the CNN-Dailymail corpus i.e. first-3 and only the first sentence for the proxy report section (AMR Bank) i.e. first-1 as they produce the best scores on the ROGUE metric compared to any other first-n. Second, we try to capture the relation between the two most important entities (we define importance by the number of occurrences of the entity in the story) of the document. For this we simply find the first sentence which contains both these entities. We call this the first co-occurrence based sentence selection. We also select the first sentence along with first co-occurrence based sentence selection as the important sentences. We call this the first co-occurrence+first based sentence selection.
As the datasets under consideration are news articles. The most important information in them is about an entity and a verb associated with it. So, to extract important information from the sentence. We try to find the entity being talked about in the sentence, we consider the most referred entity (one that occurs most frequently in the text), now for the main verb associated with the entity in the sentence, we find the verb closest to this entity in the AMR graph. We define the closest verb as the one which lies first in the path from the entity to the root.
We start by finding the position of the most referred entity in the graph, then we find the closest verb to the entity. and finally select the subtree hanging from that verb as the summary AMR.
Step 3: Summary Generation
To generate sentences from the extracted AMR graphs we can use already available generators. We use Neural AMR ( BIBREF7 ) as it provides state of the art results in sentence generation. We also use BIBREF15 (JAMR-Generator) in one of the experiments in the next section. Generators significantly effect the results, we will analyze the effectiveness of generator in the next section.
Baselines
In this section, we present the baseline models and analysis method used for each step of our pipeline.
For the CNN-Dailymail dataset, the Lead-3 model is considered a strong baseline; both the abstractive BIBREF16 and extractive BIBREF14 state-of-the art methods on this dataset beat this baseline only marginally. The Lead-3 model simply produces the leading three sentences of the document as its summary.
The key step in our pipeline is step-2 i.e. summary graph extraction.
Directly comparing the Lead-3 baseline, with AMR based pipeline to evaluate the effectiveness of step-2 is an unfair comparison because of the errors introduced by imperfect parser and generator in the AMR pipeline. Thus to evaluate the effectiveness of step-2 against Lead-3 baseline, we need to nullify the effect of errors introduce by AMR parser and generator. We achieve this by trying to introduce similar errors in the leading thre sentences of each document. We generate the AMR graphs of the leading three sentences and then generate the sentences using these AMR graph. We use parser and generator that were used in our pipeline. We consider these generated sentences as the new baseline summary, we shall now refer to it Lead-3-AMR baseline in the remaining of the paper.
For the proxy report section of the AMR bank, we consider the Lead-1-AMR model as the baseline. For this dataset we already have the gold-standard AMR graphs of the sentences. Therefore, we only need to nullify the error introduced by the generator.
Procedure to Analyze and Evaluate each step
For the evaluation of summaries we use the standard ROGUE metric. For comparison with previous AMR based summarization methods, we report the Recall, Precision and INLINEFORM0 scores for ROGUE-1. Since most of the literature on summarization uses INLINEFORM1 scores for ROGUE-2 and ROGUE-L for comparison, we also report INLINEFORM2 scores for ROGUE-2 and ROGUE-L for our method. ROGUE-1 Recall and Precision are measured for uni-gram overlap between the reference and the predicted summary. On the other hand, ROGUE-2 uses bi-gram overlap while ROGUE-L uses the longest common sequence between the target and the predicted summaries for evaluation. In rest of this section, we provide methods to analyze and evaluate our pipeline at each step.
Step-1: AMR parsing To understand the effects of using an AMR parser on the results, we compare the final scores after the following two cases- first, when we use the gold-standard AMR graphs and second when we used the AMR graphs generated by JAMR-Parser in the pipeline. Section SECREF19 contains a comparison between the two.
Step-2: Summary graph extraction For evaluating the effectiveness of the summary graph extraction step we compare the final scores with the Lead-n-AMR baselines described in section SECREF12 .
In order to compare our summary graph extraction step with the previous work ( BIBREF0 ), we generate the final summary using the same generation method as used by them. Their method uses a simple module based on alignments for generating summary after step-2. The alignments simply map the words in the original sentence with the node or edge in the AMR graph. To generate the summary we find the words aligned with the sentence in the selected graph and output them in no particular order as the predicted summary. Though this does not generate grammatically correct sentences, we can still use the ROGUE-1 metric similar to BIBREF0 , as it is based on comparing uni-grams between the target and predicted summaries.
Step-3: Generation For evaluating the quality of the sentences generated by our method, we compare the summaries generated by the first-1 model and Lead-1-AMR model on the gold-standard dataset. However, when we looked at the scores given by ROGUE, we decided to do get the above summaries evaluated by humans. This produced interesting results which are given in more detail in section SECREF20 .
Results on the Proxy report section
In table TABREF13 we report the results of using the pipeline with generation using the alignment based generation module defined in section SECREF12 , on the proxy report section of the AMR Bank. All of our methods out-perform BIBREF0 's method. We obtain best ROGUE-1 INLINEFORM0 scores using the first co-occurrence+first model for important sentences. This also out-perform our Lead-1-AMR baseline by 0.3 ROGUE-1 INLINEFORM1 points.
Effects of using JAMR Parser
In this subsection we analyze the effect of using JAMR parser for step-1 instead of the gold-standard AMR graphs. First part of table TABREF14 has scores after using the gold-standard AMR graphs. In the second part of table TABREF14 we have included the scores of using the JAMR parser for AMR graph generation. We have used the same Neural AMR for sentence generation in all methods. Scores of all methods including the Lead-1-AMR baseline have dropped significantly.
The usage of JAMR Parser has affected the scores of first co-occurrence+first and first-1 more than that for the Lead-1-AMR. The drop in ROGUE INLINEFORM0 score when we use first co-occurrence+first is around two ROGUE INLINEFORM1 points more than when Lead-1-AMR. This is a surprising result, and we believe that it is worthy of further research.
Effectiveness of the Generator
In this subsection we evaluate the effectiveness of the sentence generation step. For fair comparison at the generation step we use the gold-standard AMRs and don't perform any extraction in step-2 instead we use full AMRs, this allows to remove any errors that might have been generated in step-1 and step-2. In order to compare the quality of sentences generated by the AMR, we need a gold-standard for sentence generation step. For this, we simply use the original sentence as gold-standard for sentence generation. Thus, we compare the quality of summary generated by Lead-1 and Lead-1-AMR. The scores using the ROGUE metric are given in bottom two rows of table TABREF17 . The results show that there is significant drop in Lead-1-AMR when compared to Lead-1.
We perform human evaluation to check whether the drop in ROGUE scores is because of drop in information contained, and human readability or is it because of the inability of the ROGUE metric to judge. To perform this evaluation we randomly select ten test examples from the thirty- three test cases of the proxy report section. For each example, we show the summaries generated by four different models side by side to the human evaluators. The human evaluator does not know which summaries come from which model. A score from 1 to 10 is then assigned to each summary on the basis of readability, and information contained of summary, where 1 corresponds to the lower level and 10 to the highest. In table TABREF17 we compare the scores of these four cases as given by ROGUE along with human evaluation. The parser-generator pairs for the four cases are gold-JAMR(generator), JAMR(parser)-neural, gold-neural, and the original sentence respectively. Here gold parser means that we have used the gold-standard AMR graphs.
The scores given by the humans do not correlate with ROGUE. Human evaluators gives almost similarly scores to summary generated by the Lead-1 and Lead-1-AMR with Lead-1-AMR actually performing better on readability though it dropped some information as clear from the scores on information contained. On the other hand, ROGUE gives very high score to Lead-1 while models 1,2 and 4 get almost same scores. The similar scores of model 2 and 3 shows that generators are actually producing meaningful sentences. Thus the drop in ROGUE scores is mainly due to the inability of the ROGUE to evaluate abstractive summaries. Moreover, the ROGUE gives model 4 higher score compared to model 1 while human evaluators give the opposite scores on information contained in the sentence.
A possible reason for the inability of the ROGUE metric to properly evaluate the summaries generated by our method might be due to its inability to evaluate restructured sentences. AMR formalism tries to assign the same AMR graphs to the sentences that have same meaning so there exists a one-to-many mapping from AMR graphs to sentences. This means that the automatic generators that we are using might not be trying to generate the original sentence; instead it is trying to generate some other sentence that has the same underlying meaning. This also helps in explaining the low ROGUE-2 and ROGUE-L scores. If the sentences might be getting rephrased, they would loose most of the bi- and tri-grams from the original sentence resulting in low ROGUE-2 and ROGUE-L scores.
Analyzing the effectiveness of AMR extraction
The aim of extracting summary graphs from the AMR graphs of the sentence is to drop the not so important information from the sentences. If we are able to achieve this perfectly, the ROGUE-1 Recall scores that we are getting should remain almost the same (since we are not add any new information) and the ROGUE-1 precision should go up (as we have thrown out some useless information); thus effectively improving the overall ROGUE-1 INLINEFORM0 score. In the first two rows of table TABREF14 we have the scores after using the full-AMR and extracted AMR for generation respectively. It is safe to say that extracting the AMR results in improved ROGUE-1 precision whereas ROGUE-1 Recall reduces only slightly, resulting in an overall improved ROGUE-1 INLINEFORM1 .
Results on the CNN-Dailymail corpus
In table TABREF18 we report the results on the CNN-Dailymail corpus. We present scores by using the first-3 model. The first row contains the Lead-3-AMR baseline. The results we achieve are competitive with the Lead-3-AMR baseline. The rest of the table contains scores of Lead-3 baseline followed by the state-of-the-art method on the anonymized and non-anonymized versions of the dataset. The drop in the scores from the Lead-3(non-anonymized) to Lead-3-AMR is significant and is largely because of the error introduced by parser and generator.
Related Work
BIBREF18 showed that most of the work in text summarization has been extractive, where sentences are selected from the text which are then concatenated to form a summary. BIBREF19 transformed the input to nodes, then used the Pagerank algorithm to score nodes, and finally grow the nodes from high-value to low-value using some heuristics. Some of the approaches combine this with sentence compression, so more sentences can be packed in the summary. BIBREF20 , BIBREF21 , BIBREF22 , and BIBREF23 among others used ILPs and approximations for encoding compression and extraction.
Recently some abstractive approaches have also been proposed most of which used sequence to sequence learning models for the task. BIBREF24 , BIBREF25 , BIBREF12 , BIBREF17 used standard encoder-decoder models along with their variants to generate summaries. BIBREF26 incorporated the AMR information in the standard encoder-decoder models to improve results. Our work in similar to other graph based abstractive summarization methods BIBREF27 and BIBREF28 . BIBREF27 used dependency parse trees to produce summaries. On the other hand our work takes advantage of semantic graphs.
Need of an new Dataset and Evaluation Metric
ROGUE metric, by it is design has lots of properties that make it unsuitable for evaluating abstractive summaries. For example, ROGUE matches exact words and not the stems of the words, it also considers stop words for evaluation. One of the reasons why ROGUE like metrics might never become suitable for evaluating abstractive summaries is its incapabilities of knowing if the sentences have been restructured. A good evaluation metric should be one where we compare the meaning of the sentence and not the exact words. As we showed section SECREF20 ROGUE is not suitable for evaluating summaries generated by the AMR pipeline.
We now show why the CNN-Dailymail corpus is not suitable for Abstractive summarization. The nature of summary points in the corpus is highly extractive (Section SECREF7 for details) where most of the summary points are simply picked up from some sentences in the story. Tough, this is a good enough reason to start searching for better dataset, it is not the biggest problem with the dataset. The dataset has the property that a lot of important information is in the first few sentences and most of the summary points are directly pick from these sentences. The extractive methods based on sentence selection like SummaRunNer are not actually performing well, the results they have got are only slightly better than the Lead-3 baseline. The work doesn't show how much of the selected sentences are among the first few and it might be the case that the sentences selected by the extractive methods are mostly among the first few sentences, the same can be the problem with the abstractive methods, where most the output might be getting copied from the initial few sentences.
These problems with this corpus evoke the need to have another corpus where we don't have so much concentration of important information at any location but rather the information is more spread out and the summaries are more abstractive in nature.
Possible Future Directions
As this proposed algorithm is a step by step process we can focus on improving each step to produce better results. The most exciting improvements can be done in the summary graph extraction method. Not a lot of work has been done to extract AMR graphs for summaries. In order to make this pipeline generalizable for any sort of text, we need to get rid of the hypothesis that the summary is being extracted exactly from one sentence. So, the natural direction seems to be joining AMR graphs of multiple sentences that are similar and then extracting the summary AMR from that large graph. It will be like clustering similar sentences and then extracting a summary graph from each of these cluster. Another idea is to use AMR graphs for important sentence selection.
Conclusion
In this work we have explored a full-fledged pipeline using AMR for summarization for the first time. We propose a new method for extracting summary graph, which outperformed previous methods. Overall we provide strong baseline for text summarization using AMR for possible future works. We also showed that ROGUE can't be used for evaluating the abstractive summaries generated by our AMR pipeline.
standard ROGUE metric, Recall, Precision and INLINEFORM0 scores for ROGUE-1,  INLINEFORM2 scores for ROGUE-2 and ROGUE-L