We train multi-task autoencoders on linguistic tasks and analyze the learned hidden sentence representations. The representations change significantly when translation and part-of-speech decoders are added. The more decoders a model employs, the better it clusters sentences according to their syntactic similarity, as the representation space becomes less entangled. We explore the structure of the representation space by interpolating between sentences, which yields interesting pseudo-English sentences, many of which have recognizable syntactic structure. Lastly, we point out an interesting property of our models: The difference-vector between two sentences can be added to change a third sentence with similar features in a meaningful way.
@cite_1 use Long Short-Term Memory (LSTM) networks to encode an arbitrary sequence into a vector and decode it back into a (possibly different) sequence. They achieve results in neural machine translation (NMT) that are competitive with statistical machine translation models (SMT). The NMT objective is one of several tasks we use in our models. extend 's model with three multi-task settings: One-to-many, many-to-one and many-to-many. Their work shows that translation performance can benefit from parsing and image caption tasks. This kind of improvement from adding related tasks is also the subject of our work. note that many linguistic resources that enabled SMT are not commonly used in NMT models. They train translation models jointly with part-of-speech (POS) and named-entity recognition tasks and show that both translation and POS tagging benefit from the shared information. Our research is rooted in the same idea, but we focus on the learned representations rather than on training objectives.
@cite_1
2949888546
Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT'14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous best result on this task. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.