There have been multiple attempts with variational auto-encoders (VAE) to learn powerful global representations of complex data using a combination of latent stochastic variables and an autoregressive model over the dimensions of the data. However, for the most challenging natural image tasks the purely autoregressive model with stochastic variables still outperform the combined stochastic autoregressive models. In this paper, we present simple additions to the VAE framework that generalize to natural images by embedding spatial information in the stochastic layers. We significantly improve the state-of-the-art results on MNIST, OMNIGLOT, CIFAR10 and ImageNet when the feature map parameterization of the stochastic variables are combined with the autoregressive PixelCNN approach. Interestingly, we also observe close to state-of-the-art results without the autoregressive part. This opens the possibility for high quality image generation with only one forward-pass.
 In representation learning the goal is to learn a posterior latent distribution that explains the observed data well BID0 . Learning good representations from data can be used for various tasks such as generative modelling and semi-supervised learning (Kingma, 2013; BID14 BID14 BID23 . The decomposition of variational auto-encoders (VAE) (Kingma, 2013; BID14 provides the potential to disentangle the internal representation of the input data from local to global features through a hierarchy of stochastic latent variables. This makes the VAE an obvious candidate for learning good representations. However, in order to make inference tractable VAEs contain simplifying assumptions. This limits their ability to learn a good posterior latent representation.In complex data distributions with temporal dependencies (e.g. text, images and audio), the VAE assumption on conditional independence in the input distribution limits the ability to learn local structures. This has a significant impact on its generative performance, and thereby also the learned representations. Additionally, the one-layered VAE model with a N (0, I) latent prior poses serious constraints on the posterior complexity that the model is able to learn. A deep hierarchy of stochastic latent variables should endow the model with more expressiveness, but the VAE has a tendency to skip the learning of the higher representations since they pose a direct cost in its optimization term.There have been several attempts to eliminate the limitations of the VAE. Some concern formulating a more expressive variational distribution BID3 BID25 BID30 where other concerns learning a deeper hierarchy of latent variables . These contributions have resulted in better performance, but are still limited when modelling complex data distributions where a conditional independence does not apply. When parameterizing the VAE decoder with recurrent neural networks BID17 BID1 BID7 , the decoding architecture gets too powerful which results in unused latent stochastic variables .The limitations of the VAE have spawned interest towards other generative models such as Generative Adversarial Networks (GAN) BID8 and the autoregressive Pixel-CNN/PixelRNN models BID33 . These methods have proven powerful in learning good generative models, but the lack of stochastic latent variables makes them less suitable for representation learning purposes . Lately , we have seen several successful attempts to combine VAEs with PixelCNNs BID11 . This results Figure 1 : A visualization of FAME where the solid lines denote the variational approximation (inference/encoder/recognition) network and dashed lines denote the generative model (decoder) network for training. When performing reconstructions during training, the input image is concatenated with the output of the generative model (blue) and when generating the model follows a normal autoregressive sampling flow (red) while also using the stochastic latent variables z = z 1 , ..., z L . Both the variational approximation and the generative model follow a top-down hierarchical structure which enables precision weighted stochastic variables in the variational approximation.in a model where the global structure of the data is learned in the stochastic latent variables of the VAE and the local structure is learned in the PixelCNN. However , despite the additional complexity and potential extra expressiveness, these models do not outperform a simple autoregressive model BID32 .In this paper we present the Feature Map Variational Auto-Encoder (FAME) that combines the top-down variational approximation presented in the Ladder Variational Auto-Encoder (LVAE) ) with a spatial (feature map) representation of the stochastic latent variables and an autoregressive decoder. We show that (i) FAME outperforms previously state-of-the-art loglikelihood on MNIST, OMNIGLOT, CIFAR10 and ImageNet, (ii) FAME learns a deep hierarchy of stochastic latent variables without inactivated latent units, (iii) by removing the autoregressive decoder FAME performs close to previous state-of-the-art log-likelihood suggesting that it is possible to get good quality generation with just one forward pass. We have presented FAME, an extension to the VAE that significantly improve state-of-the-art performance on standard benchmark datasets. By introducing feature map representations in the latent stochastic variables in addition to top-down inference we have shown that the model is able to capture representations of complex image distributions while utilizing a powerful autoregressive architecture as a decoder.In order to analyze the contribution from the VAE as opposed to the autoregressive model, we have presented results without concatenating the input image when reconstructing and generating. This parameterization shows on par results with the previously state-of-the-art results without depending on the time consuming autoregressive generation.Further directions for FAME is to (i) test it on larger image datasets with images of a higher resolution, (ii) expand the model to capture other data modalities such as audio and text, (iii) combine the model in a semi-supervised framework.
We present a generative model that proves state-of-the-art results on gray-scale and natural images.
{"src": "scitldr-AIC_train", "paper_id": "Hy_o3x-0b"}