Previous work (Bowman et al., 2015; Yang et al., 2017) has found difficulty developing generative models based on variational autoencoders (VAEs) for text. To address the problem of the decoder ignoring information from the encoder (posterior collapse), these previous models weaken the capacity of the decoder to force the model to use information from latent variables. However, this strategy is not ideal as it degrades the quality of generated text and increases hyper-parameters. In this paper, we propose a new VAE for text utilizing a multimodal prior distribution, a modified encoder, and multi-task learning. We show our model can generate well-conditioned sentences without weakening the capacity of the decoder. Also, the multimodal prior distribution improves the interpretability of acquired representations. Research into generative models for text is an important field in natural language processing (NLP) and various models have been historically proposed. Although supervised learning with recurrent neural networks is the predominant way to construct generative language models BID22 BID28 BID26 , auto-regressive word-by-word sequence generation is not good at capturing interpretable representations of text or controlling text generation with global features BID1 . In order to generate sentences conditioned on probabilistic latent variables, BID1 proposed Variational Autoencoders (VAEs) BID11 for sentences. However, some serious problems that prevent training of the model have been reported.The problem that has been mainly discussed in previous papers is called "posterior collapse" BID25 . Because decoders for textual VAEs are trained with "teacher forcing" BID27 , they can be trained to some extent without relying on latent variables. As a result, the KL term of the optimization function (Equation 1) converges to zero and encoder input is ignored BID1 . Successful textual VAEs have solved this problem by handicapping the decoder so the model is forced to utilize latent variables BID1 BID30 . However, we believe that weakening the capacity of the decoder may lower the quality of generated texts and requires careful hyper-parameter turning to find the proper capacity. Therefore, we take a different approach.We focus on two overlooked problems. First, previous research fails to address the problem inherent to the structure of VAEs. The fundamental cause of posterior collapse (apart from teacher forcing) is the existence of a suboptimal local minimum for the KL term. Second, although existing models use a LSTM as the encoder, it is known that this simple model is not sufficient for text generation tasks (Bahdanau et al., 2014; BID14 BID26 . In this work, we propose a new architecture for textual VAEs with two modifications to solve these problems.First, we use a multimodal prior distribution and an unimodal posterior distribution to eliminate the explicit minima of ignoring the encoder (Chapter 3.2). Multimodal prior distributions for VAEs have been proposed recently for image and video tasks BID7 BID3 . Specifically, our model uses a Gaussian Mixture distribution as prior distribution which is trained with the method proposed by BID23 .(a ) The overall architecture of existing models.(b ) The overall architecture of our model. In the encoder, hidden states of the self-attention Encoder and BoW are concatenated. The decoder estimates BoW of the input text from the latent variables as a sub-task in addition to generating text. In our model, the prior distribution of the latent variables is a Gaussian mixture model. Second , we modify the encoder (Chapter 3.3). We empirically compare a number of existing encoders and adopt a combination of two. The first is the recently proposed method of embedding text into fixed-size variables using the attention mechanism BID12 . Although this method was originally proposed for classification tasks, we show this encoder is also effective at text generation tasks. The second is a a Bag-of-Words encoding of input text to help the encoder. It has been reported that a simple Bag-of-Words encoding is effective at embedding the semantic content of a sentence BID18 . Our experiments show that the modified encoder produces improved results only when other parts of the model are modifed as well to stabilize training. Additionally, our results imply that the self-attention encoder captures grammatical structure and Bag-of-Words captures semantic content.Finally, to help the model acquire meaningful latent variables without weakening the decoder, we add multi-task learning (Chapter 3.4). We find that a simple sub-task of predicting words included in the text significantly improves the quality of output text. It should be noted that this task does not cause posterior collapse as it does not require teacher forcing.With these modifications, our model outperforms baselines on BLEU score, showing that generated texts are well conditioned on information from the encoder (Chapter 4.3). Additionally, we show that each component of the multimodal prior distribution captures grammatical or contextual features and improves interpretability of the global features (Chapter 4.5). BID1 is the first work to apply VAEs to language modeling. They identify the problem of posterior collapse for textual VAEs and propose the usage of word dropout and KL annealing. BID16 models text as Bag-of-Words with VAEs. This is part of the motivation behind the usage of Bag-of-Words for textual VAEs. BID30 hypothesize that posterior collapse can be prevented by controlling the capacity of the decoder and propose a model with a dilated CNN decoder which allows changing the effective filter size. BID21 use a deconvolutional layer without teacher forcing to force the model into using information from the encoder. and increases hyper-parameters. We show (i) multimodal prior distribution, (ii) improvement of the encoder and (iii) multi-task learning can improve the model with a simple LSTM decoder.
We propose a model of variational autoencoders for text modeling without weakening the decoder, which improves the quality of text generation and interpretability of acquired representations.
{"src": "scitldr-AIC_train", "paper_id": "H1eZ6sRcFm"}