A Survey of the Usages of Deep Learning in Natural Language Processing <s> Morphology <s> Abstract The relationship between semantic and contextual similarity is investigated for pairs of nouns that vary from high to low semantic similarity. Semantic similarity is estimated by subjective ratings; contextual similarity is estimated by the method of sorting sentential contexts. The results show an inverse linear relationship between similarity of meaning and the discriminability of contexts. This relation, is obtained for two separate corpora of sentence contexts. It is concluded that, on average, for words in the same language drawn from the same syntactic and semantic categories, the more often two words can be substituted into the same contexts the more similar in meaning they are judged to be. <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Morphology <s> This paper describes our corpus annotation project. The annotated corpus has relevance tags which consist of predicate-argument relations, relations between nouns, and coreferences. To construct this relevance-tagged corpus, we investigated a large corpus and established the specification of the annotation. This paper shows the specification and dicult tagging problems which have emerged through the annotation so far. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Morphology <s> This paper describes a method of constructing a wide-coverage case frames from the Web. To obtain such knowledge, an enormous amount of balanced corpus is required. We consider the Web as a balanced corpus, and first build a huge text corpus from the Web. We then construct case frames from the corpus. It is infeasible to do these processes by one CPU, and so we employ a high-performance computing environment. The acquired corpus and case frames are extremely larger than previously built corpora and case frames. The resultant case frames contain most examples of usual use, and are ready to be applied to lots of NLP applications. <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Morphology <s> Unsupervised word representations are very useful in NLP tasks both as inputs to learning algorithms and as extra word features in NLP systems. However, most of these models are built with only local context and one representation per word. This is problematic because words are often polysemous and global context can also provide useful information for learning word meanings. We present a new neural network architecture which 1) learns word embeddings that better capture the semantics of words by incorporating both local and global document context, and 2) accounts for homonymy and polysemy by learning multiple embeddings per word. We introduce a new dataset with human judgments on pairs of words in sentential context, and evaluate our model on it, showing that our model outperforms competitive baselines and other neural language models. <s> BIB004 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Morphology <s> In these days, semantic analysis has been actively studied in natural language processing. For the study of semantic analysis, corpora with semantic annotations are essential. Although there are such corpora annotated on newspaper articles, there are various genres and styles, including linguistic expressions that are not found in newspaper articles. In this paper, we build a diverse document leads corpus annotated with semantic relations. To reduce the workload of annotators and annotate as many various documents as possible, we restrict the annotation target of each document to only the first three sentences. We have completed building a corpus of 1,000 documents and report the statistics of this corpus. <s> BIB005 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Morphology <s> Vector-space word representations have been very successful in recent years at improving performance across a variety of NLP tasks. However, common to most existing work, words are regarded as independent entities without any explicit relationship among morphologically related words being modeled. As a result, rare and complex words are often poorly estimated, and all unknown words are represented in a rather crude way using only one or a few vectors. This paper addresses this shortcoming by proposing a novel model that is capable of building representations for morphologically complex words from their morphemes. We combine recursive neural networks (RNNs), where each morpheme is a basic unit, with neural language models (NLMs) to consider contextual information in learning morphologicallyaware word representations. Our learned models outperform existing word representations by a good margin on word similarity tasks across many datasets, including a new dataset we introduce focused on rare words to complement existing ones in an interesting way. <s> BIB006 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Morphology <s> This paper presents a scalable method for integrating compositional morphological representations into a vector-based probabilistic language model. Our approach is evaluated in the context of log-bilinear language models, rendered suitably efficient for implementation inside a machine translation decoder by factoring the vocabulary. We perform both intrinsic and extrinsic evaluations, presenting results on a range of languages which demonstrate that our model learns morphological representations that both perform well on word similarity tasks and lead to substantial reductions in perplexity. When used for translation into morphologically rich languages with large vocabularies, our models obtain improvements of up to 1.2 BLEU points relative to a baseline system using back-off n-gram models. <s> BIB007 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Morphology <s> We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-the-art despite having 60% fewer parameters. On languages with rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model outperforms word-level/morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for language modeling. Analysis of word representations obtained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information. <s> BIB008 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Morphology <s> We present a new morphological analysis model that considers semantic plausibility of word sequences by using a recurrent neural network language model (RNNLM). In unsegmented languages, since language models are learned from automatically segmented texts and inevitably contain errors, it is not apparent that conventional language models contribute to morphological analysis. To solve this problem, we do not use language models based on raw word sequences but use a semantically generalized language model, RNNLM, in morphological analysis. In our experiments on two Japanese corpora, our proposed model significantly outperformed baseline models. This result indicates the effectiveness of RNNLM in morphological analysis. <s> BIB009 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Morphology <s> We describe an Arabic-Hebrew parallel corpus of TED talks built upon WIT3, the Web inventory that repurposes the original content of the TED website in a way which is more convenient for MT researchers. The benchmark consists of about 2,000 talks, whose subtitles in Arabic and Hebrew have been accurately aligned and rearranged in sentences, for a total of about 3.5M tokens per language. Talks have been partitioned in train, development and test sets similarly in all respects to the MT tasks of the IWSLT 2016 evaluation campaign. In addition to describing the benchmark, we list the problems encountered in preparing it and the novel methods designed to solve them. Baseline MT results and some measures on sentence length are provided as an extrinsic evaluation of the quality of the benchmark. <s> BIB010 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Morphology <s> The existing machine translation systems, whether phrase-based or neural, have relied almost exclusively on word-level modelling with explicit segmentation. In this paper, we ask a fundamental question: can neural machine translation generate a character sequence without any explicit segmentation? To answer this question, we evaluate an attention-based encoder-decoder with a subword-level encoder and a character-level decoder on four language pairs--En-Cs, En-De, En-Ru and En-Fi-- using the parallel corpora from WMT'15. Our experiments show that the models with a character-level decoder outperform the ones with a subword-level decoder on all of the four language pairs. Furthermore, the ensembles of neural models with a character-level decoder outperform the state-of-the-art non-neural machine translation systems on En-Cs, En-De and En-Fi and perform comparably on En-Ru. <s> BIB011 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Morphology <s> We present a transition-based dependency parser that uses a convolutional neural network to compose word representations from characters. The character composition model shows great improvement over the word-lookup model, especially for parsing agglutinative languages. These improvements are even better than using pre-trained word embeddings from extra data. On the SPMRL data sets, our system outperforms the previous best greedy parser (Ballesteros et al., 2015) by a margin of 3% on average. <s> BIB012
Morphology is concerned with finding segments within single words, including roots and stems, prefixes, suffixes, and-in some languages-infixes. Affixes (prefixes, suffixes, or infixes) are used to overtly modify stems for gender, number, person, et cetera. Furthermore, different stems derived from the same roots usually convey separate but related concepts. BIB006 constructed a language model that was morphologically aware. An RvNN was used to model the morphological structure of English words. A neural language model was then placed on top of the RvNN. The model was trained on the WordSim-353 dataset [Finkelstein et al. 2001] and segmentation was performed using Morfessor  ]. An additional dataset of rare words was generated, as most available datasets do a poor job of modeling these words. Two models were constructed-one using context and one not. It was found that the model that was insensitive to context over-accounted for certain morphological structures. In particular, words with the same stem were clustered together, even if they were antonyms. The context sensitive model performed better, noting the relationships between the stems, but also accounting for other features such as the prefix "un". The model was also tested on several other popular datasets BIB004 BIB001  , significantly outperforming previous embedding models on all. A good morphological analyzer is often important in the context of larger linguistic systems and other NLP tasks. As such, one recent study by Belinkov et al. [2017] examined the extent to which morphology was learned and utilized by a variety of neural machine translation models. In this study, a number of translation models were constructed, all translating from English to French, German, Czech, Arabic, or Hebrew. Encoders and decoders were LSTM-based models (some with attention mechanisms) or character aware CNNs, and the models were trained on the WIT 3 corpus BIB010  . The decoders were then replaced with part-of-speech (POS) taggers and morphological taggers, fixing the weights of the encoders to preserve the internal representations. The effects of the encoders were examined as were the effects of the decoders attached during training. The study concluded that the use of attention mechanisms decreases the performance of encoders, but increases the performance of decoders. Furthermore, it was found that character-aware models are superior to others for learning morphology and that the output language affects the performance of the encoders. Specifically, the more morphologically rich the output language, the worse the representations created by the encoders. The effects of character-level models have also been further examined in machine translation BIB011 , as well as in a number of other fields such as semantics  and parsing BIB012 . Their use has, however, been most prevalent in language modeling BIB007 BIB008 , as previously mentioned in Section 3.1.6. BIB009 analyzed a new morphological language model for unsegmented languages such as Japanese. They constructed an RNN-based model with a beam search decoder and trained it on both an automatically labeled BIB003 corpus and on a new manually labeled corpus which they presented. The model performed a number of tasks jointly, including morphological analysis, POS tagging, and lemmatization. The model was then tested on the Kyoto Text Corpus BIB002 and the Kyoto University Web Document Leads Corpus BIB005 , outperforming all baselines on all tasks.