Normalising Flows (NFs) are a class of likelihood-based generative models that have recently gained popularity. They are based on the idea of transforming a simple density into that of the data. We seek to better understand this class of models, and how they compare to previously proposed techniques for generative modeling and unsupervised representation learning. For this purpose we reinterpret NFs in the framework of Variational Autoencoders (VAEs), and present a new form of VAE that generalises normalising flows. The new generalised model also reveals a close connection to denoising autoencoders, and we therefore call our model the Variational Denoising Autoencoder (VDAE). Using our unified model, we systematically examine the model space between flows, variational autoencoders, and denoising autoencoders, in a set of preliminary experiments on the MNIST handwritten digits. The experiments shed light on the modeling assumptions implicit in these models, and they suggest multiple new directions for future research in this space. Unsupervised learning offers the promise of leveraging unlabeled data to learn representations useful for downstream tasks when labeled data is scarce BID47 , or even to generate novel data in domains where it is costly to obtain BID15 . Generative models are particularly appealing for this as they provide a statistical model of the data, x, usually in the form of a joint probability density p (x). The model's density function, its samples and representations can then be leveraged in applications ranging from semi-supervised learning and speech and (conditional) image synthesis BID44 BID30 BID14 BID26 to gene expression analysis BID13 and molecule design BID10 .In practice, data x is often high-dimensional and the optimization associated with learning p (x) can be challenging due to an abundance of local minima BID39 and difficulty in sampling from rich high-dimensional distributions BID34 . Despite this, generative modelling has undergone a surge of advancements with recent developments in likelihood-based models BID25 BID8 BID44 and Generative Adversarial Networks (GANs; BID11 ). The former class is particularly attractive, as it offers (approximate) likelihood evaluation and the ability to train models using likelihood maximisation, as well as interpretable latent representations.Autoencoders have a rich history in the unsupervised learning literature owing to their intuitive and simple construction for learning complex latent representations of data. Through fitting a parameterised mapping from the data through a lower dimensional or otherwise constrained layer back to the same data, the model learns to summarise the data in a compact latent representation. Many variants of autoencoders have been proposed to encourage the model to better encode the underlying structure of the data though regularising or otherwise constraining the model (e.g., BID38 BID1 .Denoising Autoencoders (DAEs) are a variant of the autoencoder under which noise is added to the input data that the model must then output noise-free, i.e. x = f θ (x + ) where is sampled from a, possibly structured BID48 BID49 , noise distribution ∼ q( ). They are inspired by the idea that a good representation z would be robust to noise corrupting the data x and that adding noise would discourage the model from simply learning the identity mapping. Although DAEs have been cast as generative models , sampling and computing likelihoods under the model remains challenging.Variational Autoencoders (VAEs) instead assume a probabilistic latent variable model, in which n-dimensional data x correspond to m-dimensional latent representations z following some tractable prior distribution, i.e. x ∼ p φ (x|z) with z ∼ p (z) BID25 . The task is then to learn parameters φ, which requires maximising the log marginal likelihood DISPLAYFORM0 In the majority of practical cases (e.g. p φ (x|z) taken to be a flexible neural network-conditional distribution) the above integral is intractable. A variational lower bound on the marginal likelihood is constructed using a variational approximation q θ (z|x) to the unknown posterior p (z|x): DISPLAYFORM1 The right-hand side of (2), denoted L (θ, φ), is known as the evidence lower bound (ELBO). It can be jointly optimised with stochastic optimisation w.r.t. parameters θ and φ in place of (1).Conditionals q θ (z|x) and p φ (x|z) can be viewed respectively as probabilistically encoding data x in the latent space, and reconstructing it from samples of this encoding. The first term of the ELBO encourages good reconstructions, whereas the second term encourages the model's latent variables to be distributed according to the prior p (z). Generating new data using this model is accomplished by reconstructing samples from the prior.Normalising Flows (NFs) suppose that the sought distribution p (x) can be obtained by warping a simple base density p (z), e.g. a normal distribution BID36 . They make use of the change of variables formula to obtain p (x) through a learned invertible transformation z = f θ (x) as DISPLAYFORM2 Typically, f θ : R n → R n is obtained by stacking several simpler mappings, i.e. DISPLAYFORM3 and the log-determinant obtained as the sum of log-determinants of these mappings.This formulation allows for exact maximum likelihood learning, but requires f θ to be invertible and to have a tractable inverse and Jacobian determinant. This restricts the flexibility of known transformations that can be used in NFs BID8 BID3 and leads to large and computationally intensive models in practice BID26 .NFs can also be thought of as VAEs with encoder and decoder modelled as Dirac deltas p θ (x|z) = δ (f θ (z)) and q θ (z|x) = δ f −1 θ (x) , constructed using a restricted set of transformations. Furthermore, because NFs model continuous density, to prevent trivial solutions with infinite point densities discrete data must be dequantised by adding random noise BID42 BID40 .The contribution of this work is two-fold. First, we shed new light on the relationship between DAEs, VAEs and NFs, and discuss the pros and cons of these model classes. Then, we also introduce several extensions of these models, which we collectively refer to as the Variational Denoising Autoencoders (VDAEs).In the most general form VDAEs generalise NFs and DAEs to discrete data and learned noise distributions. However, when the amount of injected noise is small, VDAE attains a form that allows for using non-invertible transformations (e.g. f θ : R n → R m , with m n). We demonstrate these theoretical advantages through preliminary experimental results on the binary and continuous versions of the MNIST dataset. We introduced Variational Denoising Autoencoders (VDAEs), a family of models the bridges the gap between VAEs, NFs and DAEs. Our model extends NFs to discrete data and non-invertible encoders that use lower-dimensional latent representations. Preliminary experiments on the MNIST handwritten digits demonstrate that our model can be successfully applied to data with discrete support, attaining competitive likelihoods and generating plausible digit samples. We also identified a failure mode of our models, in which their performance does not scale well to cases when latent and input dimensionalities are the same (i.e. when a flow-based encoder is used).Future work should address limitations of the method identified in our experiments. In particular , replacing additive coupling blocks with the more powerful invertible convolutions, affine coupling blocks and invertible residual blocks BID9 BID26 BID3 can significantly improve the variational posterior for high dimensions. It can also be interesting to explicitly condition the transformation f θ used for defining the posterior sampling procedure on the data x, for example by defining f θ (x, ) ≡ f x,θ ( ) using a hyper-network BID16 .
We explore the relationship between Normalising Flows and Variational- and Denoising Autoencoders, and propose a novel model that generalises them.
{"src": "scitldr-AIC_train", "paper_id": "HklKEUUY_E"}