Variational autoencoders (VAEs) have been successful at learning a low-dimensional manifold from high-dimensional data with complex dependencies. At their core, they consist of a powerful Bayesian probabilistic inference model, to capture the salient features of the data. In training, they exploit the power of variational inference, by optimizing a lower bound on the model evidence. The latent representation and the performance of VAEs are heavily influenced by the type of bound used as a cost function. Significant research work has been carried out into the development of tighter bounds than the original ELBO, to more accurately approximate the true log-likelihood. By leveraging the q-deformed logarithm in the traditional lower bounds, ELBO and IWAE, and the upper bound CUBO, we bring contributions to this direction of research. In this proof-of-concept study, we explore different ways of creating these q-deformed bounds that are tighter than the classical ones and we show improvements in the performance of such VAEs on the binarized MNIST dataset.
 Variational autoencoders (VAEs) BID10 , BID4 ) are powerful Bayesian probabilistic models, which combine the advantages of neural networks with those of Bayesian inference. They consist of an encoder created with a neural network architecture, which maps the high-dimensional input data, x, to a low-dimensional latent representation, z, through the posterior probability distribution, p(z|x). Then, samples from this latent distribution are decoded back to a high-dimensional signal, through another neural network architecture and the probability distribution p(x|z). Integration performed with these probability distributions from the Bayesian framework of VAEs is intractable. As a solution, variational inference is employed to perform learning in these models, whereby a tractable bound on the model evidence is optimized instead of the intractable model evidence itself BID3 . By design, the output model is set as p(x|z), usually a Bernoulli or a Gaussian probability distribution, depending on whether the target is discrete or continuous, and the prior distribution of the latent space as p(z). However, the true posterior distribution, p(z|x), remains unknown and is intractable. To solve this issue, an approximate posterior distribution, q(z|x), is learnt by means of a lower bound on the model evidence, termed the ELBO. For one data point, x (i) , writing out the Kullback-Leibler divergence between the true and approximate posterior distributions and using its positivity property yields this bound: DISPLAYFORM0 The lower bound on the model evidence, the ELBO, now becomes the cost function used during the training phase of the VAEs. Over time, the first term shows how the reconstruction loss changes and the second term how far the approximate posterior is to the prior distribution. The result of inference and the performance of VAEs on reconstructing and generating images heavily depend on the type of bound employed in training. A significant body of work has been carried out to replace the ELBO with tighter bounds on the model evidence. On the one hand, starting from an unbiased estimator of the true log-likelihood, the authors of BID0 derive an importance sampling estimate of the model evidence, the IWAE. This represents one of the tightest bounds of VAEs and has only recently been improved on in BID8 , BID11 . Increasing the number of importance samples in the IWAE objective, decreases the signal-to-noise-ratio of the gradients, which makes the learning more difficult, as the gradients suffer from a larger level of noise BID8 . Several strategies are able to correct this issue. In the first algorithm, MIWAE, the outer expectation of the IWAE objective is approximated with more than one sample, as is the case in the IWAE. The second algorithm, CIWAE, represents a convex combination of the ELBO and the IWAE bounds and the third algorithm, PIWAE, separately trains the encoder and the decoder networks with different IWAE objectives.On the other hand, leveraging different divergences between the true and the approximate posterior distributions has lead to diverse bounds on the model evidence. Starting from the Rényi α-divergence BID9 between such distributions, a family of lower and upper bounds are obtained, parameterized by α BID6 . However, these lower bounds become competitive with the IWAE, only in the limit α → −∞. In addition, the upper bounds suffer from approximation errors and bias and the means to select the best value of the hyperparameter α is unknown. Through an importance sampling scheme similar to the one found in the IWAE, these Rényi α bounds are tightened in BID15 . If the Rényi α-divergence is replaced with the χ 2 divergence, the bound on the model evidence becomes the upper bound CUBO BID1 . The Rényi α-family of bounds and others lose their interpretability as a reconstruction loss and a Kullback-Leibler divergence term that measures how close the approximate posterior is to the prior distribution. They remain just a cost function optimized during training.With different compositions of convex and concave functions, the approaches described above are unified in the K-sample generalized evidence lower bound, GLBO BID11 . This study generalizes the concept of maximizing the logarithm of the model evidence to maximizing the φ-evidence score, where φ(u) is a concave function that replaces the logarithm. It allows for great flexibility in the choice of training objectives in VAEs. One particular setting provides a lower bound, the CLBO, which surpasses the IWAE objective. We addressed the challenging task of deriving tighter bounds on the model evidence of VAEs. Significant research effort has gone in this direction, with several major contributions having been developed so far, which we reviewed in the introduction. We leveraged the q-deformed logarithm function, to explore other ways of tightening the lower bounds. As well as improvements in the estimated true log-likelihood, we found that the q-deformed bounds are much closer to the estimated true log-likelihood, than the classical bounds are. Thus, training with our novel bounds as the cost function may increase the learning ability of VAEs. Through the preliminary experiments we have conducted so far, we have achieved our goal. They show that our approach has merit and that this direction of research is worth pursuing in more depth, to produce more accurate bounds and to study their impact on the performance of VAEs.As future work, similarly to BID8 , we plan to investigate how the tightening the ELBO and the IWAE influences the learning process and affects the gradients and the structure of the latent space, compared with the classical case. In addition, we plan to explore different optimization strategies for q and to study its role in achieving tighter bounds. We will also apply our q-deformed bounds, to investigate the disentanglement problem in VAEs, see for example BID2 . The research question addressed here is how different bounds change the structure of the latent space, to provide better or worse disentanglement scores. Finally, we would also like to test our novel bounds on all the major benchmark datasets used for assessing the performance of VAEs and compare them with other state-of-the-art bounds on the model evidence.
Using the q-deformed logarithm, we derive tighter bounds than IWAE, to train variational autoencoders.
{"src": "scitldr-AIC_train", "paper_id": "rygJV8UKuV"}