We propose Automating Science Journalism (ASJ), the process of producing a press release from a scientific paper, as a novel task that can serve as a new benchmark for neural abstractive summarization. ASJ is a challenging task as it requires long source texts to be summarized to long target texts, while also paraphrasing complex scientific concepts to be understood by the general audience. For this purpose, we introduce a specialized dataset for ASJ that contains scientific papers and their press releases from Science Daily. While state-of-the-art sequence-to-sequence (seq2seq) models could easily generate convincing press releases for ASJ, these are generally nonfactual and deviate from the source. To address this issue, we improve seq2seq generation via transfer learning by co-training with new targets: (i) scientific abstracts of sources and (ii) partitioned press releases. We further design a measure for factuality that scores how pertinent to the scientific papers the press releases under our seq2seq models are. Our quantitative and qualitative evaluation shows sizable improvements over a strong baseline, suggesting that the proposed framework could improve seq2seq summarization beyond ASJ. Neural text summarization (Rush et al., 2015) has undergone an exciting evolution recently: from extractive (Nallapati et al., 2017) through abstractive (Nallapati et al., 2016) to hybrid (See et al., 2017) models; from maximum likelihood to reinforcement learning objectives (Celikyilmaz et al., 2018; Chen & Bansal, 2018) ; from small to large datasets (Grusky et al., 2018) that are also abstractive (Sharma et al., 2019) ; from short to orders of magnitude longer sources and targets (Liu et al., 2018) ; from models trained from scratch to using pre-trained representations (Edunov et al., 2019; Liu & Lapata, 2019) . Such evolution was largely supported by the emergence of seq2seq models (Cho et al., 2014; Sutskever et al., 2014) . These advances are yet to be challenged with a seq2seq summarization task that summarizes a long source to a long target with extreme paraphrasing. Below we argue that ASJ is a natural testbed for such a challenge. Science journalism is one of the few direct connections between scientific research and the general public, lead by media outlets such as Science Daily, Scientific American, and Popular Science. Their journalists face an incredibly difficult task: not only must they carefully read the scientific papers and write factual summaries, but they also need to paraphrase complex scientific concepts using a language that is accessible to the general public. To emulate what a journalist would do, we present a dataset of about 50,000 scientific papers paired with their corresponding Science Daily press releases, and we seek to train a seq2seq model to transform the former into the latter, i.e., an input scientific paper into an output popular summary. Ideally, our model would both identify and extract the relevant points in a scientific paper and it would present them in a format that a layman can understand, just as science journalists do. We now ask: would such a model be successful without a factual and accurate representation of scientific knowledge? Recent work suggests that even simple training of word embeddings could capture certain scientific knowledge from 3.3 million scientific abstracts (Tshitoya et al., 2019) . Therefore, here we propose to transfer knowledge from domains from which a seq2seq model would be able to extract factual knowledge using transfer learning (Caruana, 1997; Ruder, 2019) . We frame our approach as multitask learning (MTL). We perform co-training using both scientific abstracts and parts of the target press releases, and we view these additional domains as potential training sources for representation of scientific facts within the seq2seq model, which ideally would be helpful to ASJ. We demonstrate that MTL improves factuality in seq2seq summarization, and we measure this automatically using a novel evaluation measure that extracts random fragments of the source and evaluates the likelihood of the target given these fragments. We believe that the insights from our experiments can guide future work on a variety of seq2seq tasks. The contributions of our work can be summarized as follows: 1. We present a novel application task for seq2seq modelling: automating science journalism (ASJ). 2. We present a novel, highly abstractive dataset for summarization for the ASJ task with long source and target texts, where complex scientific notions in the source are paraphrased and explained in simple terms in the target text. 3. We propose a transfer learning approach that significantly improves the factuality of the generated summaries. 4. We propose an automatic evaluation measure that targets factuality. The rest of this paper is organized as follows: Section 2 discusses previous work. Section 3 describes the new data that we propose for ASJ. Section 4 presents our models. Section 5 introduces our transfer learning experiments for summarization. Section 6 describes our evaluation setup. Section 7 discusses our experiments and the results. Section 8 concludes and points to promising directions for future work. In this section, we focus qualitatively on the advantages and the limitations of our experiments of using transfer learning for summarization. In this work, we proposed a novel application for seq2seq modelling (ASJ), presented a highly abstractive dataset for summarization with long sources and targets (SD), proposed MTL as a means of improving factuality (AB and PART), and proposed a novel factuality-based evaluation (RA). Our transfer learning approach and our random access evaluation measure are in principle domainagnostic and hence are applicable and could be potentially useful for a variety of summarizationrelated seq2seq tasks. Our experimental results have demonstrated that MTL via special tagging for seq2seq models is a helpful step for summarization. In future work, we plan to address the limitations of the current state of AB and PART by equipping our models with pre-trained representations on large corpora, e.g., from the Web, and to use these pre-trained models as knowledge bases (Petroni et al., 2019) , thus expanding our transfer learning objectives for better factual seq2seq summarization. Ilya Sutskever, Oriol Vinayals, and Quoc V Le. Sequence to sequence learning with neural networks. In
New: application of seq2seq modelling to automating sciene journalism; highly abstractive dataset; transfer learning tricks; automatic evaluation measure.
{"src": "scitldr-AIC_train", "paper_id": "SJeRVRVYwS"}