Variational Autoencoder (VAE) is one of the most popular generative models, and enormous advances have been explored in recent years. Due to the increasing complexity of the raw data and the model architecture, deep networks are needed in VAE models while few works discuss their impacts. According to our observation, VAE does not always benefit from deeper architecture: 1) Deeper encoder makes VAE learn more comprehensible latent representations, while results in blurry reconstruction samples; 2) Deeper decoder ensures more high-quality generations, while the latent representations become abstruse; 3) When encoder and decoder both go deeper, abstruse latent representation occurs with blurry reconstruction samples at same time. In this paper, we deduce a Fisher information measure for the corresponding analysis. With such measure, we demonstrate that information loss is ineluctable in feed-forward networks and causes the previous three types of degeneration, especially when the network goes deeper. We also demonstrate that skip connections benefit the preservation of information amount, thus propose a VAE enhanced by skip connections, named SCVAE. In the experiments, SCVAE is shown to mitigate the information loss and to achieve a promising performance in both encoding and decoding tasks. Moreover, SCVAE can be adaptive to other state-of-the-art variants of VAE for further amelioration.
Recently, research on a more expressive decoder has been conducted to improve the generative performance of VAE. In the research that applies VAE to sequence modeling, a powerful decoder is proved to be more expressive . Numerous research works combine recurrent and autoregressive models to achieve a powerful decoder: @cite_5 proposed MADE, which masks the autoencoder’s parameters to respect autoregressive constraints; @cite_13 proposed a recurrent structure to gradually reconstruct observations focusing on regions of interest. @cite_1 @cite_2 model the dependencies among pixels with autoregressive density estimator , PixelCNN and PixelRNN to serve as an expressive conditional distribution.
@cite_1 @cite_5 @cite_13 @cite_2
2949899814 2952838738 1850742715 2581236139
Natural image modeling is a landmark challenge of unsupervised learning. Variational Autoencoders (VAEs) learn a useful latent representation and model global structure well but have difficulty capturing small details. PixelCNN models details very well, but lacks a latent code and is difficult to scale for capturing large structures. We present PixelVAE, a VAE model with an autoregressive decoder based on PixelCNN. Our model requires very few expensive autoregressive layers compared to PixelCNN and learns latent codes that are more compressed than a standard VAE while still capturing most non-trivial structure. Finally, we extend our model to a hierarchy of latent variables at different scales. Our model achieves state-of-the-art performance on binarized MNIST, competitive performance on 64x64 ImageNet, and high-quality samples on the LSUN bedrooms dataset. There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder's parameters to respect autoregressive constraints: each input is reconstructed only from previous inputs in a given ordering. Constrained this way, the autoencoder outputs can be interpreted as a set of conditional probabilities, and their product, the full joint probability. We can also train a single network that can decompose the joint probability in multiple different orderings. Our simple framework can be applied to multiple architectures, including deep ones. Vectorized implementations, such as on GPUs, are simple and fast. Experiments demonstrate that this approach is competitive with state-of-the-art tractable distribution estimators. At test time, the method is significantly faster and scales better than other autoregressive estimators. This paper introduces the Deep Recurrent Attentive Writer (DRAW) neural network architecture for image generation. DRAW networks combine a novel spatial attention mechanism that mimics the foveation of the human eye, with a sequential variational auto-encoding framework that allows for the iterative construction of complex images. The system substantially improves on the state of the art for generative models on MNIST, and, when trained on the Street View House Numbers dataset, it generates images that cannot be distinguished from real data with the naked eye. PixelCNNs are a recently proposed class of powerful generative models with tractable likelihood. Here we discuss our implementation of PixelCNNs which we make available at this https URL Our implementation contains a number of modifications to the original model that both simplify its structure and improve its performance. 1) We use a discretized logistic mixture likelihood on the pixels, rather than a 256-way softmax, which we find to speed up training. 2) We condition on whole pixels, rather than R G B sub-pixels, simplifying the model structure. 3) We use downsampling to efficiently capture structure at multiple resolutions. 4) We introduce additional short-cut connections to further speed up optimization. 5) We regularize the model using dropout. Finally, we present state-of-the-art log likelihood results on CIFAR-10 to demonstrate the usefulness of these modifications.