While reinforcement learning can effectively improve language generation models, it often suffers from generating incoherent and repetitive phrases paulus2017deep . In this paper, we propose a novel repetition normalized adversarial reward to mitigate these problems. Our repetition penalized reward can greatly reduce the repetition rate and adversarial training mitigates generating incoherent phrases. Our model significantly outperforms the baseline model on ROUGE-1 ,(+3.24), ROUGE-L ,(+2.25), and a decreased repetition-rate (-4.98 ).
Deep learning methods are first applied to two sentence-level abstractive summarization task on DUC-2004 and Gigaword datasets @cite_13 with an encoder-decoder model. This model is further extended by hierarchical network @cite_16 , variational autoencoders @cite_10 , a coarse to fine approach @cite_22 and minimum risk training @cite_5 . As long summaries becomes more important, CNN Daily Mail dataset was released in @cite_16 . Pointer-generator with coverage loss @cite_12 is proposed to approach the task by enabling the model to copy unknown words from article and penalizing the repetition with coverage mechanism. @cite_14 proposes deep communicating agents for representing a long document for abstractive summarization. There are more papers focusing on extractive summarizations @cite_21 @cite_0 . Memory Network @cite_6 @cite_1 , which can include external knowledge, might also be included into summarization model.
@cite_13 @cite_14 @cite_22 @cite_21 @cite_1 @cite_6 @cite_0 @cite_5 @cite_16 @cite_10 @cite_12
1843891098 2794945088 2741438349 2952138241 2735492478 2951652470 2606974598
Summarization based on text extraction is inherently limited, but generation-style abstractive methods have proven challenging to build. In this work, we propose a fully data-driven approach to abstractive sentence summarization. Our method utilizes a local attention-based model that generates each word of the summary conditioned on the input sentence. While the model is structurally simple, it can easily be trained end-to-end and scales to a large amount of training data. The model shows significant performance gains on the DUC-2004 shared task compared with several strong baselines. We present deep communicating agents in an encoder-decoder architecture to address the challenges of representing a long document for abstractive summarization. With deep communicating agents, the task of encoding a long text is divided across multiple collaborating agents, each in charge of a subsection of the input text. These encoders are connected to a single decoder, trained end-to-end using reinforcement learning to generate a focused and coherent summary. Empirical results demonstrate that multiple communicating encoders lead to a higher quality summary compared to several strong baselines, including those based on a single encoder or multiple non-communicating encoders. We present SummaRuNNer, a Recurrent Neural Network (RNN) based sequence model for extractive summarization of documents and show that it achieves performance better than or comparable to state-of-the-art. Our model has the additional advantage of being very interpretable, since it allows visualization of its predictions broken up by abstract features such as information content, salience and novelty. Another novel contribution of our work is abstractive training of our extractive model that can train on human generated reference summaries alone, eliminating the need for sentence-level extractive labels. Recently, neural models have been proposed for headline generation by learning to map documents to headlines with recurrent neural network. In this work, we give a detailed introduction and comparison of existing work and recent improvements in neural headline generation, with particular attention on how encoders, decoders and neural model training strategies alter the overall performance of the headline generation system. Furthermore, we perform quantitative analysis of most existing neural headline generation systems and summarize several key factors that impact the performance of headline generation systems. Meanwhile, we carry on detailed error analysis to typical neural headline generation systems in order to gain more comprehension. Our results and conclusions are hoped to benefit future research studies. In this work we explore deep generative models of text in which the latent representation of a document is itself drawn from a discrete language model distribution. We formulate a variational auto-encoder for inference in this model and apply it to the task of compressing sentences. In this application the generative model first draws a latent summary sentence from a background language model, and then subsequently draws the observed sentence conditioned on this latent summary. In our empirical evaluation we show that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data. Further, we explore semi-supervised compression scenarios where we show that it is possible to achieve performance competitive with previously proposed supervised models while training on a fraction of the supervised data.