Latent space based GAN methods and attention based encoder-decoder architectures have achieved impressive results in text generation and Unsupervised NMT respectively. Leveraging the two domains, we propose an adversarial latent space based architecture capable of generating parallel sentences in two languages concurrently and translating bidirectionally. The bilingual generation goal is achieved by sampling from the latent space that is adversarially constrained to be shared between both languages. First an NMT model is trained, with back-translation and an adversarial setup, to enforce a latent state between the two languages. The encoder and decoder are shared for the two translation directions. Next, a GAN is trained to generate ‘synthetic’ code mimicking the languages’ shared latent space. This code is then fed into the decoder to generate text in either language. We perform our experiments on Europarl and Multi30k datasets, on the English-French language pair, and document our performance using both Supervised and Unsupervised NMT. Neural machine translation (NMT) and neural text generation (NTG) are among the pool of successful NLP tasks handled by neural approaches. For example, NMT has acheived close to human-level performance using sequence to sequence models, which tries to solve the translation problem endto-end. NTG techniques can be categorized into three classes: Maximum Likelihood Estimation based, GAN-based and reinforcement learning (RL)-based. Recently, researchers have extensively used GANs BID8 as a potentially powerful generative model for text BID32 , because of their great success in the field of image generation.Inspired by human bilingualism, this work proposes a Bilingual-GAN agent, capable of deriving a shared latent space between two languages, and then leveraging that shared space in translation and text generation in both languages. Currently, in the literature, neural text generation (NTG) and NMT are treated as two independent problems; however, we believe that they are two sides of the same coin and could be studied jointly. Emerging latent variable-based techniques can facilitate unifying NTG and NMT and the proposed Bilingual-GAN will be a pioneering attempt in this direction.Learning latent space manifold via adversarial training has gained a lot of attention recently BID21 ; text generation and unsupervised NMT BID15 are among these examples where autoencoder (AE) latent space manifolds are learned adversarially. For NTG, in Adversarially Regularized Autoencoders (ARAE) work , a critic-generator-autoencoder combo is proposed to tackle the non-differentiability problem rising due to the discrete nature of text. The ARAE approach is to learn the continuous manifold of the autoencoder latent space and generate samples from it instead of direct synthesis of discrete (text) outputs. Output text is then reconstructed by the decoder from the generated latent samples, similarly to the autoencoding process.Adversarial learning of autoencoders' latent manifold has also been used for unsupervised NMT BID15 BID17 BID30 BID1 . In BID15 , a single denoising autoencoder is trained to derive a shared latent space between two languages using different loss functions. One of their objectives adversarially enforces the latent space generated by the encoders of the different languages to become shared and difficult to tell apart. Other objectives are autoencoder reconstruction measures and a cross-domain cost closely related to backtranslation BID24 terms.The contribution of this paper is to propose a latent space based architecture as a bilingual agent handling text generation and machine translation simultaneously. We demonstrate that our method even works when using complex multi-dimensional latent representations with attention based decoders, which weren't used in 2 RELATED WORK 2.1 LATENT SPACE BASED UNMT Neural Machine Translation BID10 BID26 BID27 constitutes the state-of-the-art in translation tasks for the majority of language pairs. On the unsupervised side, a few works BID15 ; BID0 ; BID16 have emerged recently to deal with neural machine translation without using parallel corpora, i.e sentences in one language have no matching translation in the other language. They all have a similar approach to unsupervised neural machine translation (UNMT) that uses an encoder-decoder pair sequence-to-sequence model that is shared between the languages while trying to find a latent space common to both languages. They all make use of back-translation BID24 needed for the unsupervised part of the training. BID15 use a word by word translation dictionary learned in an unsupervised way BID5 as part of their back-translation along with an adversarial loss to enforce language Independence in the latent code space. They later improve their model BID16 by removing these two elements and instead using a BPE sub-word tokenization BID23 with embeddings learned using FastText BID3 so that the sentences are embedded in a common space. BID0 have a similar flavour but uses some crosslingual embeddings to embed sentences in a shared space. They also decouple the decoder so that one is used per language. Our work proposed a novel method combining neural machine translation with word-based adversarial language generation to generate bilingual, aligned sentences. This work demonstrates the deep common grounds between language (text) generation and translation, which have not been studied before. We also explored learning a large code space comprising of the hidden states of an RNN over the entire sequence length. The results are promising and motivate a few improvements such as improving the quality of the generated sentences and eliminating language specific performance degradation. Finally, various generation methods including reinforcement learning-based, codebased, text-based and mixed methods can be incorporated into the proposed framework to improve the performance of bilingual text generation. Since during language generation our learned code space favors English sentences over French sentences, we need to remove language specific biases or explore disentangling the code space into language specific and language agnostic subspaces.
We present a novel method for Bilingual Text Generation producing parallel concurrent sentences in two languages.
{"src": "scitldr-AIC_train", "paper_id": "BJgAfh09tm"}