Sequence-to-sequence (seq2seq) neural models have been actively investigated for abstractive summarization. Nevertheless, existing neural abstractive systems frequently generate factually incorrect summaries and are vulnerable to adversarial information, suggesting a crucial lack of semantic understanding. In this paper, we propose a novel semantic-aware neural abstractive summarization model that learns to generate high quality summaries through semantic interpretation over salient content. A novel evaluation scheme with adversarial samples is introduced to measure how well a model identifies off-topic information, where our model yields significantly better performance than the popular pointer-generator summarizer. Human evaluation also confirms that our system summaries are uniformly more informative and faithful as well as less redundant than the seq2seq model. Automatic text summarization holds the promise of alleviating the information overload problem BID13 . Considerable progress has been made over decades, but existing summarization systems are still largely extractive-important sentences or phrases are identified from the original text for inclusion in the output BID22 . Extractive summaries thus unavoidably suffer from redundancy and incoherence, leading to the need for abstractive summarization methods. Built on the success of sequence-to-sequence (seq2seq) learning models BID35 , there has been a growing interest in utilizing a neural framework for abstractive summarization BID28 BID20 BID41 BID36 BID5 .Although current state-of-the-art neural models naturally excel at generating grammatically correct sentences, the model structure and learning objectives have intrinsic difficulty in acquiring semantic interpretation of the input text, which is crucial for summarization. Importantly , the lack of semantic understanding causes existing systems to produce unfaithful generations. BID3 report that about 30% of the summaries generated from a seq2seq model contain fabricated or nonsensical information.Furthermore, current neural summarization systems can be easily fooled by off-topic information. For instance , FIG0 shows one example where irrelevant sentences are added into an article about "David Collenette's resignation". Both the seq2seq attentional model BID20 and the popular pointer-generator model BID31 are particularly susceptible to unfaithful generation, partially because these models tend to rely on sentences at the beginning of the articles for summarization while being ignorant about their content. Therefore, we design a novel adversarial evaluation metric to measure the robustness of each summarizer against small amounts of randomly inserted topic-irrelevant information. The intuition is that if a summarization system truly understands the salient entities and events, it would ignore unrelated content.32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada.Article Snippet: For years Joe DiMaggio was always introduced at Yankee Stadium as "baseball's greatest living player." But with his memory joining those of Babe Ruth, Lou Gehrig, Mickey Mantle and Miller Huggins. Canada's Minister of Defense resigned today, a day after an army official testified that top military officials had altered documents to cover up responsibility for the beating death of a Somali teen-ager at the hands of Canadian peacekeeping troops in 1993. Defense minister David Collenette insisted that his resignation had nothing to do with the Somalia scandal. Ted Williams was the first name to come to mind, and he's the greatest living hitter. ... Seq2seq: George Vecsey sports of The Times column on New York State's naming of late baseball legend Joe DiMaggio as "baseball's greatest living player," but with his memory joining those of Babe Ruth, Lou Gehrig, Mickey Mantle and Miller dens. Pointer-generator: Joe DiMaggio is first name to come to mind, and Ted Williams is first name to come to mind, and he's greatest living hitter; he will be replaced by human resources minister, Doug Young, and will keep his Parliament seat for governing Liberal Party. Our Model: Former Canadian Defense Min David Collenette resigns day after army official testifies that top military officials altered documents to cover up responsibility for beating death of Somali teen-ager at hands of Canadian peacekeeping troops in 1993. To address the above issues, we propose a novel semantic-aware abstractive summarization model, inspired by the human process of writing summaries-important events and entities are first identified, and then used for summary construction. Concretely, taking an article as input, our model first generates a set of summary-worthy semantic structures consisting of predicates and corresponding arguments (as in semantic parsing), then constructs a fluent summary reflecting the semantic information. Both tasks are learned under an encoder-decoder architecture with new learning objectives. A dual attention mechanism for summary decoding is designed to consider information from both the input article and the generated predicate-argument structures. We further present a novel decoder with a segment-based reranking strategy to produce diverse hypotheses and reduce redundancy under the guidance of generated semantic information.Evaluation against adversarial samples shows that while performance by the seq2seq attentional model and the pointer-generator model is impacted severely by even a small addition of topic-irrelevant information to the input, our model is significantly more robust and consistently produces more on-topic summaries (i.e. higher ROUGE and METEOR scores for standard automatic evaluation). Our model also achieves significantly better ROUGE and METEOR scores than both models on the benchmark dataset CNN/Daily Mail BID11 . Specifically, our model's summaries use substantially fewer and shorter extractive fragments than the comparisons and have less redundancy, alleviating another common problem for the seq2seq framework. Human evaluation demonstrates that our model generates more informative and faithful summaries than the seq2seq model. Usage of Semantic Roles in Summaries. We examine the utility of the generated semantic roles. Across all models, approximately 44% of the generated predicates are part of the reference summary, indicating the adequacy of our semantic decoder. Furthermore, across all models, approximately 65% of the generated predicates are reused by the generated summary, and approximately 53% of the SRL structures are reused by the system using a strict matching constraint, in which the predicate and head words for all arguments must match in the summary. When gold-standard semantic roles are used for dual attention in place of our system generations, ROUGE scores increase by about half a point, indicating that improving semantic decoder in future work will further enhance the summaries.Coverage. We also conduct experiments using a coverage mechanism similar to the one used in BID31 . We apply our coverage in two places: (1) over the input to handle redundancy, and (2) over the generated semantics to promote its reuse in the summary. However, no significant difference is observed. Our proposed reranker handles both issues in a more explicit way, and does not require the additional training time used to learn coverage parameters.Alternative Semantic Representation. Our summarization model can be trained with other types of semantic information. For example, in addition to using the salient semantic roles from the input article, we also explore using SRL parses of the reference abstracts as training signals, but the higher level of abstraction required for semantic generation hurts performance by two ROUGE points for almost all models, indicating the type of semantic structure matters greatly for the ultimate summarization task.For future work, other semantic representation along with novel model architecture will be explored. For instance, other forms of semantic representation can be considered, such as frame semantics BID1 or Abstract Meaning Representation (AMR) BID2 . Although previous work by has shown that seq2seq models are able to successfully generate linearized tree structures, we may also consider generating semantic roles with a hierarchical semantic decoder BID34 . We presented a novel semantic-aware neural abstractive summarization model that jointly learns summarization and semantic parsing. A novel dual attention mechanism was designed to better capture the semantic information for summarization. A reranking-based decoder was proposed to promote the content coverage. Our proposed adversarial evaluation demonstrated that our model was more adept at handling irrelevant information compared to popular neural summarization models. Experiments on two large-scale news corpora showed that our model yielded significantly more informative, less redundant, and less extractive summaries. Human evaluation further confirmed that our summaries were more informative and faithful than comparisons.
We propose a semantic-aware neural abstractive summarization model and a novel automatic summarization evaluation scheme that measures how well a model identifies off-topic information from adversarial samples.
{"src": "scitldr-AIC_train", "paper_id": "rkgsd5ebjQ"}