Deep generative models such as Variational AutoEncoder (VAE) and Generative Adversarial Network (GAN) play an increasingly important role in machine learning and computer vision. However, there are two fundamental issues hindering their real-world applications: the difficulty of conducting variational inference in VAE and the functional absence of encoding real-world samples in GAN. In this paper, we propose a novel algorithm named Latently Invertible Autoencoder (LIA) to address the above two issues in one framework. An invertible network and its inverse mapping are symmetrically embedded in the latent space of VAE. Thus the partial encoder first transforms the input into feature vectors and then the distribution of these feature vectors is reshaped to fit a prior by the invertible network. The decoder proceeds in the reverse order of the encoder's composite mappings. A two-stage stochasticity-free training scheme is designed to train LIA via adversarial learning, in the sense that the decoder of LIA is first trained as a standard GAN with the invertible network and then the partial encoder is learned from an autoencoder by detaching the invertible network from LIA.   Experiments conducted on the FFHQ face dataset and three LSUN datasets validate the effectiveness of LIA for inference and generation. Deep generative models play a more and more important role in cracking challenges in computer vision as well as in other disciplines, such as high-quality image generation Karras et al., 2018a; Brock et al., 2018 ), text-to-speech transformation (van den Oord et al., 2016a; , information retrieval (Wang et al., 2017) , 3D rendering (Wu et al., 2016; Eslami et al., 2018) , and signal-to-image acquisition (Zhu et al., 2018) . Overall, the generative models fall into four categories: autoencoder and its most important variant of Variational AutoEncoder (VAE) (Kingma & Welling, 2013) , auto-regressive models (van den Oord et al., 2016b; a) , Generative Adversarial Network (GAN) (Goodfellow et al., 2014) , and normalizing flows (NF) (Tabak & Vanden-Eijnden, 2010; Tabak & Turner, 2013; Rezende & Mohamed, 2015) . Here we compare these models through the perspective of data dimensionality reduction and reconstruction. To be formal, let x be a data point in the d x -dimensional observable space R dx and y be its corresponding low-dimensional representation in the feature space R dy . The general formulation of dimensionality reduction is where f (·) is the mapping function and d y d x . The manifold learning aims at requiring f under various constraints on y (Tenenbaum1 et al., 2000; Roweis & Saul, 2000) . However, the sparsity of data points in high-dimensional space often leads to model overfitting, thus necessitating research on opposite mapping from y to x, i.e. where g(·) is the opposite mapping function with respect to f (·), to reconstruct the data. In general, the role of g(·) is a regularizer to f (·) or a generator to produce more data. The autoencoder is of mapping x f → y g →x. A common assumption in autoencoder is that the variables in lowdimensional space are usually sampled from a prior distribution P(z; θ) such as uniform or Gaussian. To differentiate from y, we let z represent the low-dimensional vector following the prior distribution. Thus we can write g : R dz → R dx , z → x = g(z), z ∼ P(z; θ). It is crucial to establish such dual maps z = f (x) and x = g(z). In the parlance of probability, the process of x → z = f (x) is called inference, and the other procedure of z → x = g(z) is called sampling or generation. VAE is capable of carrying out inference and generation in one framework by two collaborative functional modules. However, it is known that in many cases VAEs are only able to generate blurry images due to the imprecise variational inference. To see this, we write the approximation of the marginal log-likelihood where KL[q(z|x)||p(z)] is the Kullback-Leibler divergence with respect to posterior probability q(z|x) and prior p(z). This lower-bound log-likelihood usually produces imprecise inference. Furthermore, the posterior collapse frequently occurs when using more sophisticated decoder models (Bowman et al., 2015; Kingma et al., 2016 ). These two issues greatly limit the generation capability of the VAE. On the other hand, GAN is able to achieve photo-realistic generation results (Karras et al., 2018a; . However, its critical limitation is the absence of the encoder f (x) for carrying inference on real images. Effort has been made on learning an encoder for GAN under the framework of VAE, however the previous two issues of learning VAE still exist. Normalizing flows can perform the exact inference and generation with one architecture by virtue of invertible networks (Kingma & Dhariwal, 2018) . But it requires the dimension d x of the data space to be identical to the dimension d z of the latent space, thus posing computational issues due to high complexity of learning deep flows and computing the Jacobian matrices. Inspired by recent success of GANs (Karras et al., 2018a; and normalizing flows (Kingma et al., 2016; Kingma & Dhariwal, 2018) , we develop a new model called Latently Invertible Autoencoder (LIA). LIA utilizes an invertible network to bridge the encoder and the decoder of VAE in a symmetric manner. We summarize its key advantages as follows: • The symmetric design of the invertible network brings two benefits. The prior distribution can be exactly fitted from an unfolded feature space, thus significantly easing the inference problem. Besides, since the latent space is detached, the autoencoder can be trained without variational optimization thus there is no approximation here. • The two-stage adversarial learning decomposes the LIA framework into a Wasserstein GAN (only a prior needed) and a standard autoencoder without stochastic variables. Therefore the training is deterministic 2 , implying that the model will be not affected by the posterior collapse when the decoder is more complex or followed by additional losses such as the adversarial loss and the perceptual loss. • We compare LIA with state-of-the-art generative models on inference and generation/reconstruction. The experimental results on FFHQ and LSUN datasets show the LIA achieves superior performance on inference and generation. A new generative model, named Latently Invertible Autoencoder (LIA), has been proposed for generating image sample from a probability prior and simultaneously inferring accurate latent code for a given sample. The core idea of LIA is to symmetrically embed an invertible network in an autoencoder. Then the neural architecture is trained with adversarial learning as two decomposed modules. With the design of two-stage training, the decoder can be replaced with any GAN generator for high-resolution image generation. The role of the invertible network is to remove any probability optimization and bridge the prior with unfolded feature vectors. The effectiveness of LIA is validated with experiments of reconstruction (inference and generation) on FFHQ and LSUN datasets. It is still challenging to faithfully recover all the image content especially when the objects or scenes have unusual parts. For example, LIA fails to recover the hand appeared at the top of the little girl (the second row in Figure 3) . Besides, the Bombay cat's necklace (the second row in Figure 5 ) is missed in the reconstructed image. These features belong to multiple unique parts of the objects or scenes, which are difficult for the generative model to capture. One possible solution is to raise the dimension of latent variables (e.g. using multiple latent vectors) or employ the attention mechanism to highlight such unusual structures in the decoder, which is left for future work.
A new model Latently Invertible Autoencoder is proposed to solve the problem of variational inference in VAE using the invertible network and two-stage adversarial training.
{"src": "scitldr-AIC_train", "paper_id": "ryefE1SYDr"}