A Survey of the Usages of Deep Learning in Natural Language Processing <s> Relationship Extraction. <s> Attention mechanism has enhanced stateof-the-art Neural Machine Translation (NMT) by jointly learning to align and translate. It tends to ignore past alignment information, however, which often leads to over-translation and under-translation. To address this problem, we propose coverage-based NMT in this paper. We maintain a coverage vector to keep track of the attention history. The coverage vector is fed to the attention model to help adjust future attention, which lets NMT system to consider more about untranslated source words. Experiments show that the proposed approach significantly improves both translation quality and alignment quality over standard attention-based NMT. 1 <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Relationship Extraction. <s> Entity and relation extraction is a task that combines detecting entity mentions and recognizing entities' semantic relationships from unstructured text. We propose a hybrid neural network model to extract entities and their relationships without any handcrafted features. The hybrid neural network contains a novel bidirectional encoder-decoder LSTM module (BiLSTM-ED) for entity extraction and a CNN module for relation classification. The contextual information of entities obtained in BiLSTM-ED further pass though to CNN module to improve the relation classification. We conduct experiments on the public dataset ACE05 (Automatic Content Extraction program) to verify the effectiveness of our method. The method we proposed achieves the state-of-the-art results on entity and relation extraction task. (C) 2017 Elsevier B.V. All rights reserved. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Relationship Extraction. <s> In this paper, we consider the problem of open information extraction (OIE) for extracting entity and relation level intermediate structures from sentences in open-domain. We focus on four types of valuable intermediate structures (Relation, Attribute, Description, and Concept), and propose a unified knowledge expression form, SAOKE, to express them. We publicly release a data set which contains more than forty thousand sentences and the corresponding facts in the SAOKE format labeled by crowd-sourcing. To our knowledge, this is the largest publicly available human labeled data set for open information extraction tasks. Using this labeled SAOKE data set, we train an end-to-end neural model using the sequenceto-sequence paradigm, called Logician, to transform sentences into facts. For each sentence, different to existing algorithms which generally focus on extracting each single fact without concerning other possible facts, Logician performs a global optimization over all possible involved facts, in which facts not only compete with each other to attract the attention of words, but also cooperate to share words. An experimental study on various types of open domain relation extraction tasks reveals the consistent superiority of Logician to other states-of-the-art algorithms. The experiments verify the reasonableness of SAOKE format, the valuableness of SAOKE data set, the effectiveness of the proposed Logician model, and the feasibility of the methodology to apply end-to-end learning paradigm on supervised data sets for the challenging tasks of open information extraction. <s> BIB003
Another important type of information extracted from text is that of relationships. These may be possessive relationships, antonymous or synonymous relationships, or more natural relationships such as those that are familial or geographic. Note that this task is in large part a semantic task, particularly when considered in the context of neural models. Hence, discussion of underlying theory is not provided here, only a brief description of relevant approaches. One such approach is that of BIB002 , who used a bidirectional LSTM and a CNN for entity recognition as well as relationship classification. This study exhibited the usefulness of CNNs for the task. BIB003 used an attention-based GRU model with a copy mechanism. This network was novel in its use of a data structure known as a coverage mechanism BIB001 , which helped to ensure that all important information was extracted and that it was not extracted multiple times.