We train multi-task autoencoders on linguistic tasks and analyze the learned hidden sentence representations. The representations change significantly when translation and part-of-speech decoders are added. The more decoders a model employs, the better it clusters sentences according to their syntactic similarity, as the representation space becomes less entangled. We explore the structure of the representation space by interpolating between sentences, which yields interesting pseudo-English sentences, many of which have recognizable syntactic structure. Lastly, we point out an interesting property of our models: The difference-vector between two sentences can be added to change a third sentence with similar features in a meaningful way.
As an alternative to the bag-of-words feature many natural language processing (NLP) models use, propose the paragraph vector'' to represent sentences, paragraphs and whole documents. This feature outperforms bag-of-word models in text classification and sentiment analysis tasks. While it could be extended to deal with larger pieces of text, our work focuses on the sentence-level. developed a multi-task deep neural network for multi-domain classification and information retrieval, which learns general semantic representations useful for both tasks, demonstrating the advantage of multi-task learning. note that large parallel corpora for the training of NMT models are scarce. They introduce a novel system that solely relies on monolingual data while still learning to translate between languages. @cite_0 develop a generative model that connects image processing and natural language generation. Their model takes an image as input and generates an English sentence that describes the content of the image. Such a complex task relies on good, well-generalized representations, which we are exploring in this work as well.
@cite_0
2950133940
The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.