A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> Setting of the learning problem consistency of learning processes bounds on the rate of convergence of learning processes controlling the generalization ability of learning processes constructing learning algorithms what is important in learning theory?. <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> We present conditional random fields , a framework for building probabilistic models to segment and label sequence data. Conditional random fields offer several advantages over hidden Markov models and stochastic grammars for such tasks, including the ability to relax strong independence assumptions made in those models. Conditional random fields also avoid a fundamental limitation of maximum entropy Markov models (MEMMs) and other discriminative Markov models based on directed graphical models, which can be biased towards states with few successor states. We present iterative parameter estimation algorithms for conditional random fields and compare the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> In this approach to named entity recognition, a recurrent neural network, known as Long Short-Term Memory, is applied. The network is trained to perform 2 passes on each sentence, outputting its decisions on the second pass. The first pass is used to acquire information for disambiguation during the second pass. SARDNET, a self-organising map for sequences is used to generate representations for the lexical items presented to the LSTM network, whilst orthogonal representations are used to represent the part of speech and chunk tags. <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> In this paper we provide an overview of the first evaluation contest for named entity recognition in Portuguese, HAREM, which features several original traits and provided the first state of the art for the field in Portuguese, as well as a public-domain evaluation architecture. <s> BIB004 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> We describe the OntoNotes methodology and its result, a large multilingual richly-annotated corpus constructed at 90% interannotator agreement. An initial portion (300K words of English newswire and 250K words of Chinese newswire) will be made available to the community during 2007. <s> BIB005 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements. <s> BIB006 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> Large-scale linguistically annotated corpora have played a crucial role in advancing the state of the art of key natural language technologies such as syntactic, semantic and discourse analyzers, and they serve as training data as well as evaluation benchmarks. Up till now, however, most of the evaluation has been done on monolithic corpora such as the Penn Treebank, the Proposition Bank. As a result, it is still unclear how the state-of-the-art analyzers perform in general on data from a variety of genres or domains. The completion of the OntoNotes corpus, a large-scale, multi-genre, multilingual corpus manually annotated with syntactic, semantic and discourse information, makes it possible to perform such an evaluation. This paper presents an analysis of the performance of publicly available, state-of-the-art tools on all layers and languages in the OntoNotes v5.0 corpus. This should set the benchmark for future development of various NLP components in syntax and semantics, and possibly encourage research towards an integrated system that makes use of the various layers jointly to improve overall performance. <s> BIB007 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> Sentiment analysis of short texts such as single sentences and Twitter messages is challenging because of the limited contextual information that they normally contain. Effectively solving this task requires strategies that combine the small text content with prior knowledge and use more than just bag-of-words. In this work we propose a new deep convolutional neural network that exploits from characterto sentence-level information to perform sentiment analysis of short texts. We apply our approach for two corpora of two different domains: the Stanford Sentiment Treebank (SSTb), which contains sentences from movie reviews; and the Stanford Twitter Sentiment corpus (STS), which contains Twitter messages. For the SSTb corpus, our approach achieves state-of-the-art results for single sentence sentiment prediction in both binary positive/negative classification, with 85.7% accuracy, and fine-grained classification, with 48.3% accuracy. For the STS corpus, our approach achieves a sentiment prediction accuracy of 86.4%. <s> BIB008 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> Most state-of-the-art named entity recognition (NER) systems rely on handcrafted features and on the output of other NLP tasks such as part-of-speech (POS) tagging and text chunking. In this work we propose a language-independent NER system that uses automatically learned features only. Our approach is based on the CharWNN deep neural network, which uses word-level and character-level representations (embeddings) to perform sequential classification. We perform an extensive number of experiments using two annotated corpora in two different languages: HAREM I corpus, which contains texts in Portuguese; and the SPA CoNLL-2002 corpus, which contains texts in Spanish. Our experimental results shade light on the contribution of neural character embeddings for NER. Moreover, we demonstrate that the same neural network which has been successfully applied to POS tagging can also achieve state-of-the-art results for language-independet NER, using the same hyperparameters, and without any handcrafted features. For the HAREM I corpus, CharWNN outperforms the state-of-the-art system by 7.9 points in the F1-score for the total scenario (ten NE classes), and by 7.2 points in the F1 for the selective scenario (five NE classes). <s> BIB009 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> In this paper we explore a POS tagging application of neural architectures that can infer word representations from the raw character stream. It relies on two modelling stages that are jointly learnt: a convolutional network that infers a word representation directly from the character stream, followed by a prediction stage. Models are evaluated on a POS and morphological tagging task for German. Experimental results show that the convolutional network can infer meaningful word representations, while for the prediction stage, a well designed and structured strategy allows the model to outperform stateof-the-art results, without any feature engineering. <s> BIB010 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> Named entity recognition is a challenging task that has traditionally required large amounts of knowledge in the form of feature engineering and lexicons to achieve high performance. In this paper, we present a novel neural network architecture that automatically detects word- and character-level features using a hybrid bidirectional LSTM and CNN architecture, eliminating the need for most feature engineering. We also propose a novel method of encoding partial lexicon matches in neural networks and compare it to existing approaches. Extensive evaluation shows that, given only tokenized text and publicly available word embeddings, our system is competitive on the CoNLL-2003 dataset and surpasses the previously reported state of the art performance on the OntoNotes 5.0 dataset by 2.13 F1 points. By using two lexicons constructed from publicly-available sources, we establish new state of the art performance with an F1 score of 91.62 on CoNLL-2003 and 86.28 on OntoNotes, surpassing systems that employ heavy feature engineering, proprietary lexicons, and rich entity linking information. <s> BIB011 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> State-of-the-art named entity recognition systems rely heavily on hand-crafted features and domain-specific knowledge in order to learn effectively from the small, supervised training corpora that are available. In this paper, we introduce two new neural architectures---one based on bidirectional LSTMs and conditional random fields, and the other that constructs and labels segments using a transition-based approach inspired by shift-reduce parsers. Our models rely on two sources of information about words: character-based word representations learned from the supervised corpus and unsupervised word representations learned from unannotated corpora. Our models obtain state-of-the-art performance in NER in four languages without resorting to any language-specific knowledge or resources such as gazetteers. <s> BIB012 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Information Extraction <s> Information retrieval (IR) is concerned with the structure, analysis, organization, storage, searching, and dissemination of information. An IR system is designed to make a given stored collection of information items available to a user population. At one time that information consisted of stored bibliographic items, such as online catalogs of books in a library or abstracts of scientific articles. However, in today's world, the information is more likely to be full-length documents, either stored in a single location, such as newspaper archives, or available in a widely distributed form, such as the World Wide Web (WWW). <s> BIB013
Information extraction is the process of using algorithms to extract explicit or implicit information from text. The outputs of systems using these algorithms vary across implementations, but often the extracted data and the relationships within it are saved in relational databases  . Early methods included the use of simplistic information classification, pattern matching, and grammar methods to create rule-based approaches  BIB013 . Current information retrieval systems use machine learning algorithms of various kinds-supervised and unsupervised. Commonly extracted information includes named entities and relations, events and their participants, temporal information, and tuples of facts. 4.1.1 Named Entity Recognition. Named entity recognition (NER) refers to the identification of proper nouns as well as information such as dates, times, prices, and product IDs. Until recently, support vector machines BIB001 ] and conditional random fields BIB002 were the most commonly used approaches for NER. The multi-task approach of BIB006 included the task, although no results were reported. In their approach, a simple feedforward network was used, having a context with a fixed sized window around each word. Presumably, this made it difficult to capture long-distance relations between words. Furthermore, it used word embeddings as input, rendering it unable to leverage explicit character level features. Long short-term memory was first used for named entity recognition by BIB003 . The model, which was ahead of its time, had a small network due to the lack of available computing power at the time. Additionally, sophisticated numeric vector models for words were not yet available. Results obtained were slightly better than the baseline for English and much better than the baseline for German. Dos BIB009 used a deep neural network architecture, known as CharWNN, which jointly used word-level and character-level inputs to perform sequential classification. Before being tested for NER, this exact network, with the same hyper-parameters and without any hand-crafted features, was used effectively for language-independent POS tagging [Dos BIB008 . Dos BIB009 performed a number of experiments using the HAREM I annotated Portuguese corpus BIB004 , and the SPA CoNLL2002 annotated Spanish corpus  . For the Portuguese corpus, CharWNN outperformed the previous state-of-the-art system by 7.9 points in the F 1 score across ten named entity classes. It also achieved state-of-the-art performance on the Spanish corpus. The authors noted that when used alone, neither word embeddings nor character level embeddings worked as well as when used jointly. This revalidated a fact long-known in the NLP community: Joint use of word-level and character-level features is important to effective performance of named entity recognition. BIB011 were also inspired by the work of Collobert et alii. They used a bidirectional LSTM along with a character-level CNN resembling those used by dos BIB009 and BIB010 , who used it for German POS-tagging. Without using any private lexicons, detailed information about linked entities, or elaborate hand-crafted features, they were able to achieve F 1 scores of 91.62 and 86.28, respectively, on the CoNLL-2003 [Tjong  and OntoNotes BIB005 BIB007 ] datasets, forwarding the state of the art. BIB012 developed an architecture based on bidirectional LSTMs and conditional random fields. The model used both character-level inputs and word embeddings. The inputs were combined and then fed to a bidirectional LSTM, whose outputs were in turn fed to a layer that performed conditional random field (CRF) computations BIB002 . The model, when trained using dropout, obtained state-of-the-art performance in both German and Spanish. The LSTM-CRF model was also very close in both English and Dutch. The main claim of this study was that state-of-the-art results were achieved without the use any hand-engineered features or gazetteers.