Neural network models have shown excellent fluency and performance when applied to abstractive summarization. Many approaches to neural abstractive summarization involve the introduction of significant inductive bias, such as pointer-generator architectures, coverage, and partially extractive procedures, designed to mimic human summarization. We show that it is possible to attain competitive performance by instead directly viewing summarization as language modeling. We introduce a simple procedure built upon pre-trained decoder-transformers to obtain competitive ROUGE scores using a language modeling loss alone, with no beam-search or other decoding-time optimization, and instead rely on efficient nucleus sampling and greedy decoding. Neural network approaches to abstractive summarization generally encode the source document into some hidden state or representation, then decode this representation into a summarized, abstracted version of the source document [17] . These approaches usually rely on a sequence-to-sequence [20] style architecture, and tend to produce fluent, well formed natural language summaries when coupled with beam search or other decoding techniques. A weakness of traditional sequence-to-sequence learning when applied to summarization is the lack of a direct copy mechanism, leading to missing or misrepresented details in decoded summaries [2, 17] . Though attention helps ameliorate this issue by directly learning to focus on specific words or phrases in a source document [2] , many have allowed for an explicit copy mechanism inspired by Pointer Networks [22] , by optimizing a differentiable decision whether to generate new text or directly copy from the source [5, 18] . Peters et al. [15] , Devlin et al. [3] , Radford et al. [16] , and many others have shown the benefits of large-scale pretraining on large, unlabeled corpora on a variety of downstream tasks in transfer learning settings. In particular, it has been shown that large-scale, attention-only language modeling via decoder-only transformers [11] as an unsupervised pretraining task admits the ability to perform zero-shot learning on meaningful tasks involving natural language generation [16] . Motivated by this, we propose a simple method that exhibits competitive performance on abstractive summarization without using sequence-to-sequence architectures or other standard tools in the neural abstractive summarization toolbox, and instead using a decoder-only transformer language model with transfer learning. This further illustrates the utility of finetuning language models trained on open domain text. This work puts forward a simple approach to abstractive summarization by viewing sequence transduction as a language modeling problem. We show the effectiveness of using decoder-only transformers for this task, in particular, when coupled with recent advances in large-scale language modeling and transfer learning. We show that competitive performance on two benchmark datasets is possible without many of the standard tools in neural abstractive summarization, such as sequence-tosequence modeling, coverage mechanisms, direct ROUGE optimization via reinforcement learning, or beam search, instead relying on a purely language modeling loss and simple decoding mechanisms such as nucleus sampling and greedy decoding. This approach yields highly fluent text, and illustrates the power of unsupervised representation learning-based transfer learning for downstream tasks. Table 3 : Comparison of with existing methods on XSum, reported in Narayan et al. [13] .
We introduce a simple procedure to repurpose pre-trained transformer-based language models to perform abstractive summarization well.
{"src": "scitldr-AIC_train", "paper_id": "BJx5zpc58r"}