We present a new approach to defining a sequence loss function to train a summarizer by using a secondary encoder-decoder as a loss function, alleviating a shortcoming of word level training for sequence outputs. The technique is based on the intuition that if a summary is a good one, it should contain the most essential information from the original article, and therefore should itself be a good input sequence, in lieu of the original, from which a summary can be generated. We present experimental results where we apply this additional loss function to a general abstractive summarizer on a news summarization dataset. The result is an improvement in the ROUGE metric and an especially large improvement in human evaluations, suggesting enhanced performance that is competitive with specialized state-of-the-art models. Neural networks are a popular solution to the problem of text summarization, the task of taking as input a piece of natural language text, such as a paragraph or a news article, and generating a more succinct text that captures the most essential information from the original. One popular type of neural network that has achieved state of the art results is an attentional encoderdecoder neural network See et al. (2017) ; Paulus et al. (2018) ; Celikyilmaz et al. (2018) . In an encoder-decoder network, the encoder scans over the input sequence by ingesting one word token at a time to create an internal representation. The decoder is trained to compute a probability distribution over next words conditioned on a sequence prefix. A beam search decoder is typically used to find a high likelihood output sequence based on these conditional word probability distributions. Since the next word depends heavily on previous words, the decoder has little hope of a correct distribution over next words unless it has the correct previous words. Thus the decoder is typically trained using teacher forcing Williams & Zipser (1989) , where the reference sequence prefix is always given to the decoder at each decoding step. In other words, regardless of what distributions are output by the decoder in training for timesteps (1, ..., t−1), at timestep t, it is given the reference sequence prefix (y Training at the sequence level can alleviate this discrepancy, but requires a differentiable loss function. In the Related Work section we review previous efforts. We present a novel approach to address the problem by defining a loss function at the sequence level using an encoder-decoder network as a loss function. In training, the summarizer's beam search decoded output sequence is fed as input into another network called the recoder. The recoder is an independent attentional encoder-decoder trained to produce the reference summary. Our experiments show that adding the recoder as a loss function improves a general abstractive summarizer on the popular CNN/DailyMail dataset Hermann et al. (2015) ; Nallapati et al. (2016) , achieving significant improvements in the ROUGE metric and an especially large improvement in human evaluations. We have presented the use of an encoder-decoder as a sophisticated loss function for sequence outputs in the problem of summarization. The recoder allows us to define a differentiable loss function on the decoded output sequence during training. Experimental results using both ROUGE and human evaluations show that adding the recoder in training a general abstractive summarizer significantly boosts its performance, without requiring any changes to the model itself. In future work we may explore whether the general concept of using a model as loss function has wider applicability to other problems.
We present the use of a secondary encoder-decoder as a loss function to help train a summarizer.
{"src": "scitldr-AIC_train", "paper_id": "SylkzaEYPS"}