In order to learn universal sentence representations, previous methods focus on complex recurrent neural networks or supervised learning. In this paper, we propose a mean-max attention autoencoder (mean-max AAE) within the encoder-decoder framework. Our autoencoder rely entirely on the MultiHead self-attention mechanism to reconstruct the input sequence. In the encoding we propose a mean-max strategy that applies both mean and max pooling operations over the hidden vectors to capture diverse information of the input. To enable the information to steer the reconstruction process dynamically, the decoder performs attention over the mean-max representation. By training our model on a large collection of unlabelled data, we obtain high-quality representations of sentences. Experimental results on a broad range of 10 transfer tasks demonstrate that our model outperforms the state-of-the-art unsupervised single methods, including the classical skip-thoughts and the advanced skip-thoughts+LN model. Furthermore, compared with the traditional recurrent neural network, our mean-max AAE greatly reduce the training time.
Unsupervised training with unordered sentences. This kind of methods depends only on naturally occurring individual sentences. propose the paragraph vector model, which incorporates a global context vector into the log-linear neural language model @cite_11 , but at test time, inference needs to be performed to compute a new vector. propose a simple but effective Smooth Inverse Frequency (SIF) method, which represents sentence by a weighted average of word embeddings. introduce sequential denoising autoencoders (SDAE), which employ the denoising objective to predict the original source sentence given a corrupted version. They also implement bag-of-words models such as word2vec-SkipGram, word2vec-CBOW. Our model belongs to this group, which has no restriction on the required training data and can be trained on sets of sentences in arbitrary order.
@cite_11
2950133940
The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.