In this paper, we present a method for adversarial decomposition of text representation. This method can be used to decompose a representation of an input sentence into several independent vectors, where each vector is responsible for a specific aspect of the input sentence. We evaluate the proposed method on two case studies: the conversion between different social registers and diachronic language change. We show that the proposed method is capable of fine-grained con- trolled change of these aspects of the input sentence. For example, our model is capable of learning a continuous (rather than categorical) representation of the style of the sentence, in line with the reality of language use. The model uses adversarial-motivational training and includes a special motivational loss, which acts opposite to the discriminator and encourages a better decomposition. Finally, we evaluate the obtained meaning embeddings on a downstream task of para- phrase detection and show that they are significantly better than embeddings of a regular autoencoder. Despite the recent successes in using neural models for representation learning for natural language text, learning a meaningful representation of input sentences remains an open research problem. A variety of approaches, from sequence-to-sequence models that followed the work of BID37 to the more recent proposals BID2 BID29 BID8 BID25 BID36 BID5 share one common drawback. Namely, all of them encode the input sentence into just one single vector of a fixed size. One way to bypass the limitations of a single vector representation is to use an attention mechanism BID3 BID40 . We propose to approach this problem differently and design a method for adversarial decomposition of the learned input representation into multiple components. Our method encodes the input sentence into several vectors, where each vector is responsible for a specific aspect of the sentence.In terms of learning different separable components of input representation, our work most closely relates to the style transfer work, which has been applied to a variety of different aspects of language, from diachronic language differences BID42 to authors' personalities BID24 and even sentiment BID17 BID13 . The style transfer work effectively relies on the more classical distinction between meaning and form BID9 , which accounts for the fact that multiple surface realizations are possible for the same meaning. For simplicity, we will use this terminology throughout the rest of the paper.Consider the case when we encode an input sentence into a meaning vector and a form vector. We are then able to perform a controllable change of meaning or form by a simple change applied to these vectors. For example, we can encode two sentences written in two different styles, then swap the form vectors while leaving the meaning vectors intact. We can then generate new unique sentences with the original meaning, but written in a different style.In the present work, we propose a novel model for this type of decomposition based on adversarialmotivational training and design an architecture inspired by the GANs BID14 and adversarial autoencoders BID26 . In addition to the adversarial loss, we use a special motivator BID0 , which, in contrast to the discriminator, is used to provide a motivational loss to encourage the model to better decomposition of the meaning and the form, as well as specific aspects of meaning. We make all the code publicly available on GitHub 1 .We evaluate the proposed methods for learning separate aspects of input representation on the following case studies:1. Learning to separate out a representation of the specific diachronic slice of language. One may express the same meaning using the Early Modern English (e.g. What would she have?) and the contemporary English ( What does she want?)2. Learning a representation for a social register BID16 -that is, subsets of language appropriate in a given context or characteristic of a certain group of speakers. These include formal and informal language, the language used in different genres (e.g., fiction vs. newspapers vs. academic texts), different dialects, and even literary idiostyles. We experiment with the registers corresponding to the titles of scientific papers vs. newspaper articles. The classifier used in the transfer strength metric achieves very high accuracy (0.832 and 0.99 for the Shakespeare and Headlines datasets correspondingly). These results concur with the results of BID34 and BID13 , and show that the two forms in the corpora are significantly different.Following BID13 , we show the result of different configuration of the size of the form and meaning vectors on FIG2 . Namely, we report combinations of 64 and 256-dimensional vectors.Note that the sizes of the form vector are important. The larger is the form vector, the higher is the transfer strength, but smaller is content preservation. This is consistent with BID13 , where they observed a similar behaviour.It is clear that the proposed method achieves significantly better transfer strength then the previously proposed model. It also has a lower content preservation score, which means that it repeats fewer exact words from the source sentence. Note that a low transfer strength and very high (0.9) content preservation score means that the model was not able to successfully learn to transfer the form and the target sentence is almost identical to the source sentence. The Shakespeare dataset is the hardest for the model in terms of transfer strength, probably because it is the smallest dataset, but the proposed method performs consistently well in transfer of both form and meaning and, in contrast to the baseline.Fluency of generated sentences Note that there is no guarantee that the generated sentences would be coherent after switching the form vector. In order to estimate how this switch affects the fluency of generated sentences, we trained a language model on the Shakespeare dataset and calculated the perplexity of the generated sentences using the original form vector and the average of form vectors of k random sentences from the opposite style (see subsubsection 5.1.1). While the perplexity of such sentences does go up, this change is not big (6.89 vs 9.74).
A method which learns separate representations for the meaning and the form of a sentence
{"src": "scitldr-AIC_train", "paper_id": "SJzwb2RcK7"}