We introduce a model for constructing vector representations of words by composing characters using bidirectional LSTMs. Relative to traditional word representation models that have independent vectors for each word type, our model requires only a single vector per character type and a fixed set of parameters for the compositional model. Despite the compactness of this model and, more importantly, the arbitrary nature of the form-function relationship in language, our "composed" word representations yield state-of-the-art results in language modeling and part-of-speech tagging. Benefits over traditional baselines are particularly pronounced in morphologically rich languages (e.g., Turkish).
The work in @cite_46 proposes a window-based model for word labeling. Given a vocabulary of output labels @math , the model predicts the probability of a given label @math , for word @math as @math , where @math are the input words indexed from @math to @math . The model projects each word @math into a @math -dimentional vector @math . Then, for the word to be labelled @math , the representations @math and the @math closest words @math are concatenated into a @math -dimensional vector and a linear combination is applied using the parameters in matrix @math where @math is defined as an hyperparameter that determines the dimensionality of the output of this operation. Then, a non-linearity is applied to the resulting vector, yielding vector @math . The output label can be predicted by applying a linear transformation to this vector parametrized by @math , which generates a vector with the size of the label vocabulary @math , and a softmax is applied to this vector to obtain a probability distribution over @math .
@cite_46
2158899491
We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.