A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. Training method <s> We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. Training method <s> Continuous space language models have recently demonstrated outstanding results across a variety of tasks. In this paper, we examine the vector-space word representations that are implicitly learned by the input-layer weights. We find that these representations are surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset. This allows vector-oriented reasoning based on the offsets between words. For example, the male/female relationship is automatically learned, and with the induced vector representations, “King Man + Woman” results in a vector very close to “Queen.” We demonstrate that the word vectors capture syntactic regularities by means of syntactic analogy questions (provided with this paper), and are able to correctly answer almost 40% of the questions. We demonstrate that the word vectors capture semantic regularities by using the vector offset method to answer SemEval-2012 Task 2 questions. Remarkably, this method outperforms the best previous systems. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. Training method <s> Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic, but the origin of these regularities has remained opaque. We analyze and make explicit the model properties needed for such regularities to emerge in word vectors. The result is a new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods. Our model efficiently leverages statistical information by training only on the nonzero elements in a word-word cooccurrence matrix, rather than on the entire sparse matrix or on individual context windows in a large corpus. The model produces a vector space with meaningful substructure, as evidenced by its performance of 75% on a recent word analogy task. It also outperforms related models on similarity tasks and named entity recognition. <s> BIB003
Before feeding the training data to the model, one pre-step is to transfer the words to vectors, which makes a proper form that the neural network could receive. Usually, the most frequent V words in one language will be chosen, and each language generally has different word set. Despite that the embedding weights will be learned in the training period, the pre-trained word embedding vector such as word2vec BIB001 BIB002 or Glove vector BIB003 can also be applied directly. In the training period, this model is fed by a bilingual corpus for Encoder and Decoder. The learning objective is to map the input sequence with the corresponding sequence in the target language correctly. Like other DNN models, the input sentence pair is embedded as a list of word vectors, and the model parameters are initialized randomly. The training process could be formulated as trying to updating its parameters periodically until getting the minimum loss of the neural network. In the implementation, RNN will refine the parameters after it processes a subset of data that contain a batch of training samples; this subset is called the mini-batch set. To simplify the discussion of the training process, we take one sentence pair (one training sample) as example. For the Encoder, the encoding RNN will receive one word in source sentence once a time-step. After several steps, all words will be compressed into the hidden state of the Encoder. Then the final vector will be transferred to the Decoder. For Decoder, the input comes from two sources: the thought vector that is directly sent to Decoder, and the correct word in the last time-step (the first word is < EOS >). The output process in Decoder can be seen as a reverse work of Encoder; Decoder predicts one word in each time-step until the last symbol is < EOS >.