The goal of generative models is to model the underlying data distribution of a
 sample based dataset. Our intuition is that an accurate model should in principle
 also include the sample based dataset as part of its induced probability distribution.
 To investigate this, we look at fully trained generative models using the Generative
 Adversarial Networks (GAN) framework and analyze the resulting generator
 on its ability to memorize the dataset. Further, we show that the size of the initial
 latent space is paramount to allow for an accurate reconstruction of the training
 data. This gives us a link to compression theory, where Autoencoders (AE) are
 used to lower bound the reconstruction capabilities of our generative model. Here,
 we observe similar results to the perception-distortion tradeoff (Blau & Michaeli
 (2018)). Given a small latent space, the AE produces low quality and the GAN
 produces high quality outputs from a perceptual viewpoint. In contrast, the distortion
 error is smaller for the AE. By increasing the dimensionality of the latent
 space the distortion decreases for both models, but the perceptual quality only
 increases for the AE. Generative Adversarial Networks (GANs) were introduced by Goodfellow et al. (2014) for the purpose of generative modelling. Since then this framework has been successfully applied to works in style transfer by Karras et al. (2018) , superresolution by Shocher et al. (2018) and semi-supervised learning by Salimans et al. (2016) , but what GANs actually learn is still poorly understood as has been noted by Webster et al. (2019) . Recently, GANs have been used to solve inverse problems, where it was tried to use the generated manifold to solve an auxiliary task like image completion (Webster et al. (2019) ), MRI reconstruction (Narnhofer et al. (2019) ) or anomaly detection (Shocher et al. (2018) ). For those applications, it is necessary to know if the generator NN actually describes the distribution well. Related works have shown that faithfully reconstructing the images from a generator network is not trivial (Webster et al. (2019) ; Shmelkov et al. (2018) ). The original convergence proof by Goodfellow et al. (2014) assumes that the generator and discriminator Neural Networks (NNs) have infinite capacity and they showed that the discriminator network models the Jensen-Shannon divergence between the probability distribution induced by the generator and the real data distribution. Others have adapted this paradigm and devised loss functions which have been shown to converge to other divergences or distances on the underlying probability distributions ; Nowozin et al. (2016) ; Mao et al. (2017) ). Regularization techniques like Gradient Penalty (Gulrajani et al. (2017) ) and Spectral Norm (Miyato et al. (2018) ) did improve the stability of the training process ) but it is still unclear how well this NNs actually approximate such distances even for trivial problems (Pinetz et al. (2018) ). Additionally, it is not at all clear how the generated distribution or the actual target distribution look like. Arora & Zhang (2017) used the birthday paradox to empirically gauge the size of the support of the generated distribution. GANs are used to transform a well understood low dimensional distribution (in practice either gaussian or uniform) to a high dimensional unknown distribution (Goodfellow et al. (2014) ) by playing a min-max game between two NNs. This paper is based around the intuition, that an estimated probability distribution from a dataset X has high precision if a high percentage of the actual data samples are included in the estimated distribution. To have a sense of what an adequate capacity for a generator network is, we use AE to reconstruct the dataset first. This work relies on the assumption that it is easier to reconstruct the data samples alone, than to reconstruct the entire manifold and Section 5 shows empirical evidence for this. Based on our intuition the manifold consists of the data samples and imposes additional structure between the data samples. In contrast by just reproducing the data samples, no such additional restrictions are given, making the problem strictly simpler. AEs can be trained rapidly and have been researched in detail for a long time (Rumelhart et al. (1985) ). In contrast, trying to do a hyperparameter search on the GAN networks themselves gives rise to all kinds of problems, like instabilities in the training process, random failures and dependence on random seeds for their performance ). Hence, our contributions are as follows: • An investigation of the impact of the dimensionality of the latent space on the generated manifold. We showcase that the fit of the data depends heavily on the latent space. We also show similar results thereby to the perception-distortion tradeoff (Blau & Michaeli (2018) ), where with a small dimension for the latent space, the GAN optimizes for perception and the AE optimizes for distortion. • Relating the GAN problem to a compression task and furthermore using compression tools via deep learning to produce a lower bound for a dataset dependent suitable dimensionality of the latent space. • An investigation of the generated manifold and the limitations thereof to produce shifted or noisy images and how this relates to the size of the latent space and overfitting of the generative model. The remainder of this paper is organized as follows. Section 2 shows the related work. Then in Section 3 we revisit the theory behind pseudo inverting NNs and we explain our methodology in Section 4. In Section 5 the results are shown. Section 6 draws conclusions based on those results. In this work, we show that by reducing the problem to a compression task, we can give a lower bound on the required capacity and latent space dimensionality of the generator network for the distribution estimation task. Relating these two different algorithms to each other, the literature surrounding AEs for invertability and dimensionality reduction, as well as the corresponding theoretical advancements are used. While in this work the encoder and the discriminator NNs use the same architecture, we have not discovered any relation between them. Still, the same architecture works well empirically for both task. Using our framework we show various properties of generator networks. The perceptual image quality appears to be independent of the actual size of the latent space, which is in contrast to AE, where the visual quality improves if the dimensionality of the latent space is increased. However, the ability to reconstruct the training set correctly does depend on the initial latent space. Also the ability of the generator to reconstruct deviations from the original dataset, like a validation set or shifted images depends just as much on the initial latent space. However, the same cannot be said for reconstructing arbitrary noise images. Here the reconstruction ability is independent of the initial latent space unless it is chosen very large, suggesting that the generator has learned realistic natural image features. Here for smaller latent spaces we can still observe face like features. Our hypothesis is that the implicit bias induced by the generator architecture lends itself to generating natural images and GANs are skilled at that by learning primitives which can be combined to construct arbitrary images. In future works we want to use our setup to search towards better and more reliable generators for images.
We analyze the impact of the latent space of fully trained generators by pseudo inverting them.
{"src": "scitldr-AIC_train", "paper_id": "Hygy01StvH"}