Inspired by the success of self attention mechanism and Transformer architecture
 in sequence transduction and image generation applications, we propose novel self
 attention-based architectures to improve the performance of adversarial latent code-
 based schemes in text generation. Adversarial latent code-based text generation
 has recently gained a lot of attention due to their promising results. In this paper,
 we take a step to fortify the architectures used in these setups, specifically AAE
 and ARAE. We benchmark two latent code-based methods (AAE and ARAE)
 designed based on adversarial setups. In our experiments, the Google sentence
 compression dataset is utilized to compare our method with these methods using
 various objective and subjective measures. The experiments demonstrate the
 proposed (self) attention-based models outperform the state-of-the-art in adversarial
 code-based text generation. Text generation is of particular interest in many natural language processing (NLP) applications such as dialogue systems, machine translation, image captioning and text summarization. Recent deep learning-based approaches to this problem can be categorized into three classes: auto-regressive or maximum likelihood estimation (MLE)-based, generative adversarial network (GAN)-based and reinforcement learning (RL)-based approaches. BID26 ) model the text (language) as an auto-regressive process, commonly using RNNs. RNNs compactly represent the samples history in the form of recurrent states. In these models, text is generated by predicting next token (character, word, etc) based on the previously generated ones BID9 . The results of objective and subjective evaluations are presented in Tables 2 to 5. As seen, the proposed self attention-based (SALSA) architectures consistently outperform the non-attention-based benchmarks in terms of diversity (measured by reverse perplexity). Moreover, they often show better performance in terms of output quality (measured by BLEU, self BLEU, preplexity and human evaluations) on the long and complicated sentences of the GSC dataset.As seen in the generated samples TAB0 , human evaluation TAB3 ) and objective metrics (Tables 2 to 4), the original AAE and ARAE setups perform very poorly on GSC with long sentences. With reverse perplexities of over 8000 and high self-BLEU scores close to 0.9, they suffer from a high level of mode collapse (repeated sentences).Human evaluations do not account for lack of diversity. The reason is humans are presented with a number of shuffled sentences and asked to evaluate them independently (without knowing which sentence coming from which model). Hence, in our experiments for the original AAE and ARAE, a model can generate similar sentences (maybe due to mode collapse) and still receives high subjective scores. It seems that, in our experiments, the original ARAE model suffers from mode collapse. We can see that it has slightly higher human evaluation scores, but extremely poor diversity metrics, i.e. very high reverse perplexity and self-BLEU scores. It can also be seen in the randomly selected generated sentences TAB0 , where all the sentences start with "A man" and invariably mention he is being arrested or accused of grievous crimes. This is likely because the sentences in the GSC dataset are long and that their structure is elaborate. SALSA-ARAE on the other hand reliably produces sentences of quality with great diversity.SALSA-AAE has both considerably higher individual quality metrics than the original AAE and much better diversity metrics. It is the strongest pure adversarial text model. As seen in TAB3 , SALSA-AAE provides the best grammaticality, semantic consistency and Fluency performance. In this paper, we introduced SALSA-TEXT, a Transformer-based architecture for adversarial codebased text generation. It incorporates self-attention mechanism by utilizing Transformer architecture in autoencoder and GAN setups. Our extensive experiments demonstrate the better performance of our models compared to the state-of-the-art in adversarial code-based text generation (without self-attention). The proposed architectures provide diverse, long and high quality output sentences as confirmed by objective metrics and human evaluations in extensive experiments.As a future direction, it is beneficial to study the performance of self attention in other text generation methods including variational code-based and reinforcement learning-based approaches. Another interesting direction is to experiment with deeper Transformer-based autoencoders to better capture the underlying language model and perform unsupervised pre-training isnpired by the success of BID0 and Radford et al..
We propose a self-attention based GAN architecture for unconditional text generation and improve on previous adversarial code-based results.
{"src": "scitldr-AIC_train", "paper_id": "B1liWh09F7"}