This work introduces a simple network for producing character aware word embeddings. Position agnostic and position aware character embeddings are combined to produce an embedding vector for each word. The learned word representations are shown to be very sparse and facilitate improved results on language modeling tasks, despite using markedly fewer parameters, and without the need to apply dropout. A final experiment suggests that weight sharing contributes to sparsity, increases performance, and prevents overfitting. When processing text for Natural Language Processing (NLP), one important decision to make is how to represent the words for a given model or system. For many tasks tackled by deep learning such as language modeling, language understanding, and translation, the use of word embeddings has become the standard approach. BID20 BID4 BID0 This is in part due to their ability to represent complex syntactic and semantic relationships between words as spatial relationships within the embedding dimensions BID13 .Embeddings are generally implemented as a lookup table for computational efficiency. However for those unfamiliar with their use it may be beneficial to formulate them as the output of the first layer in a neural network. This is true for a layer that has one-hot feature vectors as inputs, no bias, and no activation function.For a given one-hot feature vector x, the activations of such a layer can be computed by xW , which is equivalent to selecting the row W i of the weight matrix, where x i == 1. The weight matrix or embedding lookup matrix can then be optimized via typical techniques such as gradient descent, including from subsequent layers of a DNN through back propagation. BID15 For word embeddings , the basic approach is to assign a unique vector of trainable parameters to each word in a vocabulary. These vectors are referred to in this paper as token embeddings. Token embeddings present a number of limitations. For example, any out-of-vocabulary word cannot be represented except as a pre-defined < U N K > token. A corollary of this is that the number of embeddings (and therefore trainable parameters) grows linearly with the size of the vocabulary. Furthermore, characters are ignored, meaning that potentially useful morphological information is thrown out.To get around these limitations, researchers have explored building word embeddings from lower level character representations. A variety of techniques have been presented, including the use of feedforward multi layer perceptrons (MLPs) BID3 , convolutional neural networks (CNNs) BID9 BID10 , and recurrent neural networks (RNNs) BID12 . These character level representations of words have the advantage over token embeddings of allowing an open vocabulary, usually having fewer parameters, and improving performance by making use of information available in sub-word level features.The most successful approaches for building word embeddings from characters use CNNs. BID9 However, the architecture of CNNs is designed to identify positioninvariant features, not the specific ordering of characters that make up a word's spelling. Here we ask whether such ordering is a valuable source of information.A number of convolutional features of varying size can be used to capture some ordering, for example within each feature independently. However as the vocabulary is expanded, the number convolutional features must be increased to compensate BID9 . Once convolution is performed, the used of a deep highway network , as introduced by BID18 , is then needed to produce the final word embedding.The current study presents a simple fully connected architecture for combining characters. In this framework, each character is represented both by position-agnostic character embeddings and position-aware character embeddings, which we call spelling embeddings. The combination of the two allows the model to learn both position invariant features and positional features. A word embedding is then constructed by combining both the character and spelling embeddings of the word, for example by summing or by averaging them together. The resulting vector is then passed through a nonlinear MLP that combines the character and spelling information to produce the final word embedding.This MLP, along with the spelling and character embeddings, were trained via gradient descent as inputs to a Recurrent Neural Network (RNN) being trained for a language modeling task. Results show that including the spelling information facilitates improvement over token embeddings despite requiring far fewer parameters. Without the position information, character embeddings alone are not sufficient in this fully connected architecture.An analysis of the learned representations at the word embedding level shows much greater sparsity for spelling embeddings than for token embeddings, and demonstrates some of the negative impacts of dropout on the representations. Finally, we compare token based models with a fully connected layer of shared weights to raw token embeddings with no weight sharing. Passing the token embeddings through a layer of shared weights is shown to drastically increase representation sparsity and prevent overfitting. Given that the character and spelling weights are heavily shared among word embeddings, this is presented as possible explanation for the spelling aware model's robustness against overfitting. This work shows that a simple fully connected network is able to produce character aware word embeddings that outperform traditional token embeddings. The architecture is relatively simple compared to previous approaches that use CNNs or RNNs to combine character information. This work lacks a direct comparison to these other character aware methods, which is an obvious direction for future work.Investigation into the word embeddings produced by the presented architectures reveal a number of interesting properties. Spelling embeddings are especially resistant to overfitting compared to token embeddings, and are also significantly more sparse in their activations. Furthermore, dropout is shown to have some negative impacts on the word representations, and weight sharing is presented as a better way to regularize word embeddings.Spelling embeddings exhibit the most weight sharing, because each character embedding is shared across many words in the vocabulary. This may be a contributing factor to their increased sparsity and resistance to overfitting. Additional evidence for this is provided in the comparison of raw token embeddings to those passed through a fully connected layer.Whereas raw token embeddings share none of their weights with other words in the vocabulary, token embeddings passed through a fully connected layer share all the weights in that layer across the entire vocabulary. Not only do token embeddings enjoy increased resistance to overfitting when passed though a shared weight layer, they also become drastically more sparse. Whereas dropout is a popular technique for regularization in NLP, it can have a negative impact on the word embeddings, causing some of them to gain a Gini coefficient of 0. This suggests that these particular words have completely homogeneous representations and are indistinguishable from each other.On the smaller dataset the number of shared parameters in the fully connected layer of the token embeddings is large compared to the vocabulary size. In this case, dropout is needed to prevent overfitting. On the larger dataset, the number of shared parameters is much smaller relative to the vocabulary size. In this case dropout is not needed for the token embeddings and actually hinders them. The spelling embeddings perform worse with dropout on both datasets.The architecture presented here should be compared to the state of the art character CNN results obtained on the One Billion Word benchmark. BID9 ) Also, whereas a number hyper-parameters governing the number and size of the layers were tried before the ones presented in this paper were found, other techniques such as highway networks BID18 have not yet been investigated.Furthermore, extending the concept of character aware word embeddings to the output softmax layer is another open area of research that has been tried with character CNNs BID9 , but not to our knowledge with a spelling network as presented in this work.
A fully connected architecture is used to produce word embeddings from character representations, outperforms traditional embeddings and provides insight into sparsity and dropout.
{"src": "scitldr-AIC_train", "paper_id": "rJ8rHkWRb"}