We propose a new framework for abstractive text summarization based on a sequence-to-sequence oriented encoderdecoder model equipped with a deep recurrent generative decoder ( DRGN ) . Latent structure information implied in the target summaries is learned based on a recurrent latent random model for improving the summarization quality . Neural variational inference is employed to address the intractable posterior inference for the recurrent latent variables . Abstractive summaries are generated based on both the generative latent variables and the discriminative deterministic states . Extensive experiments on some benchmark datasets in different languages show that DRGN achieves improvements over the state-ofthe-art methods . Automatic summarization is the process of automatically generating a summary that retains the most important content of the original text document ( Edmundson , 1969 ; Luhn , 1958 ; Nenkova and McKeown , 2012 ) . Different from the common extraction-based and compression-based methods , abstraction-based methods aim at constructing new sentences as summaries , thus they require a deeper understanding of the text and the capability of generating new sentences , which provide an obvious advantage in improving the focus of a summary , reducing the redundancy , and keeping a good compression rate ( Bing et al . , 2015 ; Rush et al . , 2015 ; Nallapati et al . , 2016 ) . * The work described in this paper is supported by a grant from the Grant Council of the Hong Kong Special Administrative Region , China ( Project Code : 14203414 ) . Some previous research works show that human-written summaries are more abstractive ( Jing and McKeown , 2000 ) . Moreover , our investigation reveals that people may naturally follow some inherent structures when they write the abstractive summaries . To illustrate this observation , we show some examples in Figure 1 , which are some top story summaries or headlines from the channel " Technology " of CNN . After analyzing the summaries carefully , we can find some common structures from them , such as " What " , " What-Happened " , " Who Action What " , etc . For example , the summary " Apple sues Qualcomm for nearly $ 1 billion " can be structuralized as " Who ( Apple ) Action ( sues ) What ( Qualcomm ) " . Similarly , the summaries " [ Twitter ] [ fixes ] [ botched @POTUS account transfer ] " , " [ Uber ] [ to pay ] [ $ 20 million ] for misleading drivers " , and " [ Bipartisan bill ] aims to [ reform ] [ H-1B visa system ] " also follow the structure of " Who Action What " . The summary " The emergence of the ' cyber cold war " ' matches with the structure of " What " , and the summary " St. Louis ' public library computers hacked " follows the structure of " What-Happened " . Intuitively , if we can incorporate the latent structure information of summaries into the abstractive summarization model , it will improve the quality of the generated summaries . However , very few existing works specifically consider the latent structure information of summaries in their summarization models . Although a very popular neural network based sequence-to-sequence ( seq2seq ) framework has been proposed to tackle the abstractive summarization problem ( Lopyrev , 2015 ; Rush et al . , 2015 ; Nallapati et al . , 2016 ) , the calculation of the internal decoding states is entirely deterministic . The deterministic transformations in these discriminative models lead to limitations on the representation ability of the latent structure information . Miao and Blunsom ( 2016 ) extended the seq2seq framework and proposed a generative model to capture the latent summary information , but they did not consider the recurrent dependencies in their generative model leading to limited representation ability . To tackle the above mentioned problems , we design a new framework based on sequenceto-sequence oriented encoder-decoder model equipped with a latent structure modeling component . We employ Variational Auto-Encoders ( VAEs ) ( Kingma and Welling , 2013 ; Rezende et al . , 2014 ) as the base model for our generative framework which can handle the inference problem associated with complex generative modeling . However , the standard framework of VAEs is not designed for sequence modeling related tasks . Inspired by ( Chung et al . , 2015 ) , we add historical dependencies on the latent variables of VAEs and propose a deep recurrent generative decoder ( DRGD ) for latent structure modeling . Then the standard discriminative deterministic decoder and the recurrent generative decoder are integrated into a unified decoding framework . The target summaries will be decoded based on both the discriminative deterministic variables and the generative latent structural information . All the neural parameters are learned by back-propagation in an end-to-end training paradigm . The main contributions of our framework are summarized as follows : ( 1 ) We propose a sequence-to-sequence oriented encoder-decoder model equipped with a deep recurrent generative decoder ( DRGD ) to model and learn the latent structure information implied in the target summaries of the training data . Neural variational inference is employed to address the intractable posterior inference for the recurrent latent variables . ( 2 ) Both the generative latent structural information and the discriminative deterministic variables are jointly considered in the generation process of the abstractive summaries . ( 3 ) Experimental results on some benchmark datasets in different languages show that our framework achieves better performance than the state-of-the-art models . We propose a deep recurrent generative decoder ( DRGD ) to improve the abstractive summarization performance . The model is a sequenceto-sequence oriented encoder-decoder framework equipped with a latent structure modeling component . Abstractive summaries are generated based on both the latent variables and the deterministic states . Extensive experiments on benchmark
Although humans follow inherent structures in summary writing, currently there are no abstractive summarization models which take latent structure information and recurrent dependencies into account.
They propose a Variational Auto-Encoder-based sequence-to-sequence oriented encoder-decoder model with a deep recurrent generative decoder which learns latent structure information implied in the target summaries.
The proposed model outperforms the state-of-the-art models on some datasets in different languages.