One of the challenges in training generative models such as the variational auto encoder (VAE) is avoiding posterior collapse. When the generator has too much capacity, it is prone to ignoring latent code. This problem is exacerbated when the dataset is small, and the latent dimension is high. The root of the problem is the ELBO objective, specifically the Kullback–Leibler (KL) divergence term in objective function. This paper proposes a new  objective function to replace the KL term with one that emulates the maximum mean discrepancy (MMD) objective. It also introduces a new technique, named latent clipping, that is used to control distance between samples in latent space. A probabilistic autoencoder model, named $\mu$-VAE, is designed and trained on MNIST and MNIST Fashion datasets, using the new objective function and is shown to outperform models trained with ELBO and $\beta$-VAE objective. The $\mu$-VAE is less prone to posterior collapse, and can generate reconstructions and new samples in good quality. Latent representations learned by $\mu$-VAE are shown to be good and can be used for downstream tasks such as classification.   Autoencoders(AEs) are used to learn low-dimensional representation of data. They can be turned into generative models by using adversarial, or variational training. In the adversarial approach, one can directly shape the posterior distribution over the latent variables by either using an additional network called a Discriminator (Makhzani et al., 2015) , or using the encoder itself as a discriminator (Huang et al., 2018) . AEs trained with variational methods are called Variational Autoencoders (VAEs) (Kingma & Ba, 2014; Rezende et al., 2014) . Their objective maximizes the variational lower bound (or evidence lower bound, ELBO) of p θ (x). Similar to AEs, VAEs contain two networks: Encoder -Approximate inference network: In the context of VAEs, the encoder is a recognition model q φ (z|x) 1 , which is an approximation to the true posterior distribution over the latent variables, p θ (z|x). The encoder tries to map high-level representations of the input x onto latent variables such that the salient features of x are encoded on z. Decoder -Generative network: The decoder learns a conditional distribution p θ (x|z) and has two tasks: i) For the task of reconstruction of input, it solves an inverse problem by taking mapped latent z computed using output of encoder and predicts what the original input is (i.e. reconstruction x ≈ x). ii) For generation of new data, it samples new data x , given the latent variables z. During training, encoder learns to map the data distribution p d (x) to a simple distribution such as Gaussian while the decoder learns to map it back to data distribution p(x) 2 . VAE's objective function has two terms: log-likelihood term (reconstruction term of AE objective function) and a prior regularization term 3 . Hence, VAEs add an extra term to AE objective function, and approximately maximizes the log-likelihood of the data, log p(x), by maximizing the evidence lower bound (ELBO): Maximizing ELBO does two things: • Increase the probability of generating each observed data x. • Decrease distance between estimated posterior q(z|x) and prior distribution p(z), pushing KL term to zero. Smaller KL term leads to less informative latent variable. Pushing KL terms to zero encourages the model to ignore latent variable. This is especially true when the decoder has a high capacity. This leads to a phenomenon called posterior collapse in literature (Razavi et al., 2019; Dieng et al., 2018; van den Oord et al., 2017; Bowman et al., 2015; Sønderby et al., 2016; Zhao et al., 2017) . This work proposes a new method to mitigate posterior collapse. The main idea is to modify the KL term of the ELBO such that it emulates the MMD objective (Gretton et al., 2007; Zhao et al., 2019) . In ELBO objective, minimizing KL divergence term pushes mean and variance parameters of each sample at the output of encoder towards zero and one respectively. This , in turn, brings samples closer, making them indistinguishable. The proposed method replaces the KL term in the ELBO in order to encourage samples from latent variable to spread out while keeping the aggregate mean of samples close to zero. This enables the model to learn a latent representation that is amenable to clustering samples which are similar. As shown in later sections, the proposed method enables learning good generative models as well as good representations of data. The details of the proposal are discussed in Section 4.
This paper proposes a new  objective function to replace KL term with one that emulates maximum mean discrepancy (MMD) objective. 
{"src": "scitldr-AIC_train", "paper_id": "rJgWiaNtwH"}