A Survey of the Usages of Deep Learning in Natural Language Processing <s> Summarization <s> Excerpts of technical papers and magazine articles that serve the purposes of conventional abstracts have been created entirely by automatic means. In the exploratory research described, the complete text of an article in machine-readable form is scanned by an IBM 704 data-processing machine and analyzed in accordance with a standard program. Statistical information derived from word frequency and distribution is used by the machine to compute a relative measure of significance, first for individual words and then for sentences. Sentences scoring highest in significance are extracted and printed out to become the "auto-abstract." <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Summarization <s> ●● ● ● ● To summarize is to reduce in complexity, and hence in length, while retaining some of the essential qualities of the original. This paper focusses on document extracts, a particular kind of computed document summary. Document extracts consisting of roughly 20% of the original cart be as informative as the full text of a document, which suggests that even shorter extracts may be useful indicative summmies. The trends in our results are in agreement with those of Edmundson who used a subjectively weighted combination of features as opposed to training the feature weights using a corpus. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Summarization <s> The importance of a Web page is an inherently subjective matter, which depends on the readers interests, knowledge and attitudes. But there is still much that can be said objectively about the relative importance of Web pages. This paper describes PageRank, a mathod for rating Web pages objectively and mechanically, effectively measuring the human interest and attention devoted to them. We compare PageRank to an idealized random Web surfer. We show how to efficiently compute PageRank for large numbers of pages. And, we show how to apply PageRank to search and to user navigation. <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Summarization <s> We present a novel graph-based summarization framework (Opinosis) that generates concise abstractive summaries of highly redundant opinions. Evaluation results on summarizing user reviews show that Opinosis summaries have better agreement with human summaries compared to the baseline extractive method. The summaries are readable, reasonably well-formed and are informative enough to convey the major opinions. <s> BIB004 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Summarization <s> We introduce a stochastic graph-based method for computing relative importance of textual units for Natural Language Processing. We test the technique on the problem of Text Summarization (TS). Extractive TS relies on the concept of sentence salience to identify the most important sentences in a document or set of documents. Salience is typically defined in terms of the presence of particular important words or in terms of similarity to a centroid pseudo-sentence. We consider a new approach, LexRank, for computing sentence importance based on the concept of eigenvector centrality in a graph representation of sentences. In this model, a connectivity matrix based on intra-sentence cosine similarity is used as the adjacency matrix of the graph representation of sentences. Our system, based on LexRank ranked in first place in more than one task in the recent DUC 2004 evaluation. In this paper we present a detailed analysis of our approach and apply it to a larger data set including data from earlier DUC evaluations. We discuss several methods to compute centrality using the similarity graph. The results show that degree-based methods (including LexRank) outperform both centroid-based methods and other systems participating in DUC in most of the cases. Furthermore, the LexRank with threshold method outperforms the other degree-based techniques including continuous LexRank. We also show that our approach is quite insensitive to the noise in the data that may result from an imperfect topical clustering of documents. <s> BIB005 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Summarization <s> Summarization based on text extraction is inherently limited, but generation-style abstractive methods have proven challenging to build. In this work, we propose a fully data-driven approach to abstractive sentence summarization. Our method utilizes a local attention-based model that generates each word of the summary conditioned on the input sentence. While the model is structurally simple, it can easily be trained end-to-end and scales to a large amount of training data. The model shows significant performance gains on the DUC-2004 shared task compared with several strong baselines. <s> BIB006 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Summarization <s> Many natural language processing applications use language models to generate text. These models are typically trained to predict the next word in a sequence, given the previous words and some context such as an image. However, at test time the model is expected to generate the entire sequence from scratch. This discrepancy makes generation brittle, as errors may accumulate along the way. We address this issue by proposing a novel sequence level training algorithm that directly optimizes the metric used at test time, such as BLEU or ROUGE. On three different tasks, our approach outperforms several strong baselines for greedy generation. The method is also competitive when these baselines employ beam search, while being several times faster. <s> BIB007 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Summarization <s> The prevalent approach to sequence to sequence learning maps an input sequence to a variable length output sequence via recurrent neural networks. We introduce an architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training and optimization is easier since the number of non-linearities is fixed and independent of the input length. Our use of gated linear units eases gradient propagation and we equip each decoder layer with a separate attention module. We outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT'14 English-French translation at an order of magnitude faster speed, both on GPU and CPU. <s> BIB008
Summarization is the task of finding elements or characteristics of interest from documents in order to produce an encapsulation of the most important information. There are two primary types of summarization techniques: extractive and abstractive. The first focuses on sentence extraction, simplification, reordering, and concatenation to relay the important information contained in documents using text taken directly from the documents. A large number of extractive summarization algorithms have been proposed over the years. These include frequency-based approaches  BIB001 ; machine-learning (naive Bayes) -based algorithms BIB002 ; and graph-based algorithms, computing centrality measure and relative importance of sentences BIB005 using the PageRank algorithm BIB003 . Abstractive summaries rely on expressing documents' contents through generation-style abstraction, possibly using words never seen in the documents  . Historically, abstractive summarization algorithms have included graph-based algorithms BIB004 as well as parse tree and graph-to-graph transformations . More recently-and with increasing success-deep learning methods have been used for abstractive summarization. Deep learning approaches generally use recurrent encoder-decoder architectures. Breaking rank was the application to summarization of the feedforward neural language model by BIB006 . The language model used an attention-based encoder and a generative beam search decoder. The initial input was given directly to both the language model and the convolutional attention-based encoder, which determined contextual importance surrounding the summary sentences and phrases. The performance of the model was comparable to other state-of-the-art models of the time. As in other areas, attention mechanisms have improved performance of encoder-decoder models. A state-of-the-art approach developed by Paulus et al. [2017] used a multiple intra-temporal attention encoder mechanism that considered not only the input text tokens, but also the output tokens used by the decoder for previously generated words. They also used similar hybrid crossentropy loss functions to those proposed by BIB007 , which led to decreases in training and execution by orders of magnitude. Finally, they recommended using strategies seen in reinforcement learning to modify gradients and reduce exposure bias, which has been noted in models trained exclusively via supervised learning. The use of attention also boosted accuracy in the fully convolutional model proposed by BIB008 , who implemented an attention mechanism for each layer.