The goal of generative models is to model the underlying data distribution of a sample based dataset. Our intuition is that an accurate model should in principle also include the sample based dataset as part of its induced probability distribution. To investigate this, we look at fully trained generative models using the Generative Adversarial Networks (GAN) framework and analyze the ing generator on its ability to memorize the dataset. Further, we show that the size of the initial latent space is paramount to allow for an accurate reconstruction of the training data. This gives us a link to compression theory, where Autoencoders (AE) are used to lower bound the reconstruction capabilities of our generative model. Here, we observe similar to the perception-distortion tradeoff . Given a small latent space, the AE produces low quality and the GAN produces high quality outputs from a perceptual viewpoint. In contrast, the distortion error is smaller for the AE. By increasing the dimensionality of the latent space the distortion decreases for both models, but the perceptual quality only increases for the AE. Generative Adversarial Networks (GANs) were introduced by for the purpose of generative modelling. Since then this framework has been successfully applied to works in style transfer by , superresolution by and semi-supervised learning by , but what GANs actually learn is still poorly understood as has been noted by. Recently, GANs have been used to solve inverse problems, where it was tried to use the generated manifold to solve an auxiliary task like image completion , MRI reconstruction or anomaly detection . For those applications, it is necessary to know if the generator NN actually describes the distribution well. Related works have shown that faithfully reconstructing the images from a generator network is not trivial . The original convergence proof by assumes that the generator and discriminator Neural Networks (NNs) have infinite capacity and they showed that the discriminator network models the Jensen-Shannon divergence between the probability distribution induced by the generator and the real data distribution. Others have adapted this paradigm and devised loss functions which have been shown to converge to other divergences or distances on the underlying probability distributions;; ). Regularization techniques like Gradient Penalty and Spectral Norm did improve the stability of the training process ) but it is still unclear how well this NNs actually approximate such distances even for trivial problems . Additionally, it is not at all clear how the generated distribution or the actual target distribution look like. used the birthday paradox to empirically gauge the size of the support of the generated distribution. GANs are used to transform a well understood low dimensional distribution (in practice either gaussian or uniform) to a high dimensional unknown distribution by playing a min-max game between two NNs. This paper is based around the intuition, that an estimated probability distribution from a dataset X has high precision if a high percentage of the actual data samples are included in the estimated distribution. To have a sense of what an adequate capacity for a generator network is, we use AE to reconstruct the dataset first. This work relies on the assumption that it is easier to reconstruct the data samples alone, than to reconstruct the entire manifold and Section 5 shows empirical evidence for this. Based on our intuition the manifold consists of the data samples and imposes additional structure between the data samples. In contrast by just reproducing the data samples, no such additional restrictions are given, making the problem strictly simpler. AEs can be trained rapidly and have been researched in detail for a long time . In contrast, trying to do a hyperparameter search on the GAN networks themselves gives rise to all kinds of problems, like instabilities in the training process, random failures and dependence on random seeds for their performance ). Hence, our contributions are as follows: • An investigation of the impact of the dimensionality of the latent space on the generated manifold. We showcase that the fit of the data depends heavily on the latent space. We also show similar thereby to the perception-distortion tradeoff , where with a small dimension for the latent space, the GAN optimizes for perception and the AE optimizes for distortion. • Relating the GAN problem to a compression task and furthermore using compression tools via deep learning to produce a lower bound for a dataset dependent suitable dimensionality of the latent space. • An investigation of the generated manifold and the limitations thereof to produce shifted or noisy images and how this relates to the size of the latent space and overfitting of the generative model. The remainder of this paper is organized as follows. Section 2 shows the related work. Then in Section 3 we revisit the theory behind pseudo inverting NNs and we explain our methodology in Section 4. In Section 5 the are shown. Section 6 draws based on those . GANs have been invented by for the task of image synthesis and semisupervised learning. The initial formulation has been shown to suffer from mode collapse and vanishing gradient problems and multiple other loss functions have been proposed (; ; . We choose the Wasserstein GAN, due to it using the L2-distance as its transport cost function to judge the quality of the generated manifold, which is similar to the Mean Squared Error (MSE) used in the classical AE . To satisfy the Lipschitz constraint we use , due to its good performance in review works; ). For evaluation purposes we need a quality metrics for GAN and the most common ones rely on the usage of classification models, which are trained on ImageNet . Two commonly used metrics are the Inception Score and the Fréchet Inception Distance (FID) . To combat the bias of having trained on proposed to train a new classifier to evaluate the GAN model, but this is time consuming and prone to errors by not having a standardized metric. We opted for the FID score, due to its prevalence in related works; ). AEs have been studied extensively in the past and initially have been proposed by. They have found usage in compression literature (Cheng et al. (2018a; b); ). Their first usage in a GAN framework has been shown in the Energy based GAN by. Since then they have been used as preprocessing to reduce the dimensionality of the data , as regularizers or as generative models themselves . The inversion of generator networks has been initially used by and there have been multiple works on how to improve their method by adding heuristics like stochastic clipping on top of that. We opted against stochastic clipping, because we already project back to the likely set of candidates. Stochastic clipping would add even more randomness to the . Generalization of GANs through the detection of overfitting has been proposed recently by. Therein the authors are mostly concerned (f) CelebA: dim(z) = 5000 Figure 1: Histogram of the values in the weight matrix and the convolutional layers of a trained WGAN-GP with a latent space specific dimensionality for CIFAR-10 and CelebA datasets. Notice, that the weights share the same characteristics as if they were drawn from a normal distribution. with detecting memorisation of the training set in the manifold and avoid said memorisation to generalize. In contrast we make the assumption that an estimated probability distribution based on samples should include those samples and overfitting is detected based on a hold out set as well as the ability to reconstruct that hold out set as well as the training set. Additionally, we show that the ability of the GAN to memorize depends mostly on the dimensionality of the latent space and the capacity of the generator network and can be estimated using compression techniques. In this work, we want to show the fit of the distribution of trained GAN newtworks using inversions of such networks. This motivates the question of " When are NNs invertible?". Due to the nonconvex and non-bijective nature of NNs using ReLU activations invertibility is not given in the general case. The function induced by the NN is in general neither injective nor surjective and it is trivial to construct such cases. However, if the weights are gaussian distributed, proofs exist, which guarantee that the NN is invertible with a high probability. For this, we revisit the inversion theory of NNs largely pioneered by. They empirically showed that the weights of trained NNs actually exhibit random like properties, enabling the usage of Random Matrix Theory (RMT) to investigate such networks. Theorem 1 in proves that we can invert 2-layer NNs with high probability. In their work they empirically showed that this also holds for multi layer networks. In Figure 1 we show that the same holds true for trained GAN networks. Note, that this property holds across datasets and latent spaces used to train those networks. Glorot initialization was used to initialize these networks. Therefore, the standard deviation used for the initialization process differs for each layer. However, as is demonstrated in Fig. 1, all layers converged to a similar gaussian distribution. Another indicator, that the optimization process is likely to succeed is given by a trick proposed by for classification networks. It allows us to visualize the loss surface of the optimization problem for one sample at a time. For a particular data point z *, we chose two orthogonal vectors α, β and plot the following 2D function: We chose α = z * − z 0 to model the optimization path towards the original data point. For this experiment we do not need to run an optimization, because the point is to show that it is possible There is a clear optimization path from z 0 = 0 to a point z * even in this simplified plot, empirically validating the claims of , that an accurate reconstruction of the generator manifold is possible works using first order methods. to recover points from the manifold. To ensure that β is an orthogonal vector, we simply chose this vector by setting β = 0 and then. Then α, β are orthogonal and we can use them as a basis for the plots shown in Fig. 1. For visualization purposes, we scaled β to have the same norm as α; e.g. ||β|| 2 = ||α|| 2. We vary a, b ∈ [−2, 2]. In our experiments this always ed in benign structures as is shown in Fig. 2. If it is trivial to recover points from the manifold and impossible to recover the training images, than this implies that the manifold does not include the training images. Currently, designing new GAN networks works by running a huge hyperparameter search on a plausible set of combinations; ). noted that the progress in GANs has actually been quite minimal and vastly overstated, and depends on the hyperparameters more than on the improvements in the methodology, while did notice that non-saturating loss in combiniation with spectral norm produced consistently good , but never the best ones. The goal of the GAN framework is to encapsulate the high dimensional image distribution in a lower dimensional noise distribution, whereas a generator network maps the lower dimensional distribution to the former high dimensional image distribution . This implies that we compress the larger distribution onto the simple lower dimensional distribution and use the generator network as an approximation for the decompression. We rely on the assumption, that less capacity is needed to reconstruct the training set, than to reconstruct the entire manifold. Therefore, to gain some insight into the necessary capacity of the generator network, the usage of an AE is proposed to gauge the number of parameters and the size of the latent space to enable suitable approximation of the training set. In theory, a latent dimension of 1 would be sufficient to reconstruct the entire dataset, by mapping a curve to the datapoints. A simple smooth function that does this for a dataset {x 1, .., x n} = X ∈ R n×d and {p 1, ..., p n} = P ∈ R n×1 looks like this: This function will output the corresponding image x i for every p i. In practice, this behavior is not observed. AEs with small latent spaces tend to output blurry approximations of the training images as is shown in Section 5. Once a suitable AE for a dataset is found, the decoder part of the AE is used as the generative model of choice. The AE is trained using the L2-loss, which also is the transport cost function used in WGANs ) to gauge the quality of the generated manifold. Therefore, the objective function of the AE using a decoder d and an encoder e is defined as follows: ||d(e(x)) − x|| 2 We do not opt for any regularization, because we want to fit the data as closely as possible to determine if the generator capacity is adequate for reconstructing the dataset. In this work we show empirically in Sec. 5 that if the AE already fails to produce crisp , then also the GAN algorithm fails at having the dataset included in the generated manifold. One thing to note is that, while the encoder part is used as our discriminator, it has a completely different objectives and it is unclear, if a better encoder is also a better discriminator. The key part of this work is to analyse the ability of the GAN to memorise the training dataset, following the assumption, that a probablity distribution based on a sample dataset has to include that dataset. To our best knowledge there is no analytical way to invert generative NNs. Therefore this ill posed problem is stated as an inverse optimzation problem for a function g: This is solved using a standard first order gradient method. However, as is shown in Section 5 the ing latent vectors are unlikely to stem from the targeted noise distribution, if there are no constraints added. Therefore, the search space for z is restricted to the space of plausible latent vectors. Plausibility in this context means that the probability of the latent vector originating from this space is larger than 99.7% (3-sigma-rule). In the standard multivariate gaussian distribution the norm of the vector z corresponds inversely to the likelihood of z ∈ N (0, I). Therefore we restrict the norm to be ||z|| ≤ E[||z||] + 3 V AR [||z||]. For the multivariate gaussian distribution this is easy to calculate and the final objective function is given as follows: This is solved using a projected ADAM optimizer . Our experiments are done on medium sized datasets (CIFAR-10 and CelebA (64 × 64 resolution)) and the proof of concept on MNIST is relegated to the supplementary material. The NN adapted for this task is the standard DCGAN from. Convolutional layers and resize operations are used instead of transposed convolutions, to avoid checkerboard artifacts . The full hyperparameter setting is shown in the supplementary material. The visual quality measured by the FID of the ing samples is on par with the survey work by. This work is not aimed at improving the visual quality of the samples generated by GAN algorithms. For CIFAR-10 the test set is used as our validation set and for CelebA we randomly split off 20% of the dataset to be used as validation set. In principle, getting from an AE to a GAN is just a rearrangement of the NNs. For the GAN training, the default hyperparameter are used as in. The AE NNs are optimized using the ADAM optimizer with default hyperparameters for all the datasets, always on a single GPU. A comparison of the ing reconstruction errors is shown in Fig. 3. Therein the different AE and GAN models only differ in their latent space sizes. Increasing the latent space dimension also increases the number of parameters. However, for the sake of simplicity this was neglected. The AE network acts as a lower bound for the GAN algorithm, therefore validating our intuition that the AE complexity lower-bounds the GAN complexity. The visual quality of the generated images is evaluated using the FID versus the reconstructed images by an AE for the same latent space sizes. The experimental are shown in Fig. 4 for the CIFAR and CelebA datasets and demonstrate, that the visual quality of the GAN generated images barely changes with different latent dimensionality. In contrast, the AE reconstructions gain quality and sharpness as the latent space increases. This reflects the output of those models shown in Fig. 5, where the ing images are clearly an approximation of the actual training images. On the other hand, the GAN reconstructions have similar visual quality independent of the actual latent space. However, the ing images are only remotely similar to the actual image for insufficiently large latent spaces. This phenomenon has been observed in other works, where a similar looking person to an actual training image is found by the inversion procedure . However, we show in Fig. 5, that this phenomenon depends mostly on the size of the latent space. curve for different latent spaces (lower is better). Increasing or decreasing the latent space does not in a noticeable difference in the quality metric for the WGAN-GP, but it does for the AE. While it is possible to reconstruct the dataset accurately using a large enough latent space, in this section we look at what else those models can reconstruct. In these experiments, our baseline model The first columns show the original images. As we increase the latent space of the autoencoder the image quality increasing. The GAN reconstructions are of high perceptual quality even for low dimensional spaces, but correspond to other people as has been observed by. is a WGAN-GP trained on CelebA. The first experiment shown in Fig. 6 uses translated training images. This is especially challenging for this dataset, due to the pre-aligned face images used in the CelebA dataset. As is shown in Fig. 6 the error goes up and the visual quality of the reconstructions deteriorates even for small pixel shifts as long as the latent space is small enough. Figure 6: Impact of translation on the reconstruction ability of GAN networks. Notice, that by increasing latent space dimension, the generator network is able to reconstruct translations with a similar precision as the original images. The second experiment is on reconstructing the CIFAR-10 dataset using the CelebA trained model. To provide the same resolution, the CIFAR-10 images were upscaled to 64 × 64 using bilinear interpolation. If the latent space is large enough, we can reconstruct the images well, as shown in Fig. 7. The reconstruction of the CelebA images is always worse as measured by the MSE (Fig. 7). The final experiment is on reconstructing noise images. For this a dataset of uniform noise images with size 64 × 64 is used. The are shown in Fig. 8. As is suggested by the , it is in fact not possible to reconstruct everything. One thing to note though is that with lower latent spaces some face structure is visible in the reconstructed images, showing that those models have a strong prior on the face structure. We noticed and the corresponding experiment is shown in the supplementary material, that the norm of the latent vector is smaller than of random vectors. Our hypothesis is that those images are the base face features, which are refined with larger latent vectors, but this guess needs further validation work. In this work, we show that by reducing the problem to a compression task, we can give a lower bound on the required capacity and latent space dimensionality of the generator network for the distribution estimation task. Relating these two different algorithms to each other, the literature surrounding AEs for invertability and dimensionality reduction, as well as the corresponding theoretical advancements are used. While in this work the encoder and the discriminator NNs use the same architecture, we have not discovered any relation between them. Still, the same architecture works well empirically for both task. Using our framework we show various properties of generator networks. The perceptual image quality appears to be independent of the actual size of the latent space, which is in contrast to AE, where the visual quality improves if the dimensionality of the latent space is increased. However, the ability to reconstruct the training set correctly does depend on the initial latent space. Also the ability of the generator to reconstruct deviations from the original dataset, like a validation set or shifted images depends just as much on the initial latent space. However, the same cannot be said for reconstructing arbitrary noise images. Here the reconstruction ability is independent of the initial latent space unless it is chosen very large, suggesting that the generator has learned realistic natural image features. Here for smaller latent spaces we can still observe face like features. Our hypothesis is that the implicit bias induced by the generator architecture lends itself to generating natural images and GANs are skilled at that by learning primitives which can be combined to construct arbitrary images. In future works we want to use our setup to search towards better and more reliable generators for images. For our experiments a standard CNN was used as is described in Table 1. All the convolutions use 5 × 5 kernels. For the CIFAR-10 experiments we removed one of the up and downlayers and started with 1024 feature maps. We used the ADAM optimizer with learning rate 1E − 4 and β 1 = 0. and β 2 = 0.9 similar to the original WGAN-GP as described by. We trained for 80.000 generator iterations and 5 discriminator iterations per generator update for all of our networks. We also trained the AE for 80.000 iterations with ADAM using the default hyperparameters. The inversion of the generator networks was done using 10.000 ADAM steps using default hyperparameters. We found that the changes afterwards were negligible. The initial experiments on MNIST validating the legitimacy of the approach are shown in Fig. 9. Instead of using a visual quality metric to determine if the ing images are similar enough, we used the reconstructed training images of the GAN for training a LeNet network to determine if the images are descriptive. The manifold of MNIST has been faithfully learned before by. In contrast to their work, we show that this only works given a large enough latent space and that the reconstruction error of the AE and the WGAN-GP converge for larger latent spaces. In this section the main question is if the found reconstructions likely to appear in the generative model. For this reason the inverse optimization procedure is run without the projection onto the feasible set for dim(z) = 1000 on CIFAR-10. The ing latent vector norm of the reconstructions is shown in Fig 10. The corresponding set of vectors, which is likely to appear in the training is shown in blue, while the training set reconstructions are shown in red and the test set reconstructions are shown in green. As is apparent therein, the overlap is virtually non-existant and this is the reason why we used a projected gradient method. We also investigated the difference between the ability of GAN to reconstruct images of CIFAR-10 while trained on CelebA compared to AE. The are shown in Fig. 11 and as is already shown in Fig. 7 the GAN algorithm is better at reconstructing CIFAR-10 images than CelebA images. However, this is not the case for the AE. And as a final experiment we show visual on CIFAR-10 images produced by an inverted GAN and by an AE in Fig. 12. As in the main paper we can the same behavior again. The AE produces initially blurry and then sharp ones and the GAN produces sharp of unrelated images and then produces images which are closer to the actual images. The first column are the actual images. As we increase the latent space of the autoencoder we can see the image quality increasing. The same cannot be said about the GAN reconstructions. Those are of high quality even for low dimensional space, but correspond to other people.
We analyze the impact of the latent space of fully trained generators by pseudo inverting them.