Generative adversarial networks (GANs) have been extremely effective in approximating complex distributions of high-dimensional, input data samples, and substantial progress has been made in understanding and improving GAN performance in terms of both theory and application. 
 However, we currently lack quantitative methods for model assessment. Because of this, while many GAN variants being proposed, we have relatively little understanding of their relative abilities. In this paper, we evaluate the performance of various types of GANs using divergence and distance functions typically used only for training. We observe consistency across the various proposed metrics and, interestingly, the test-time metrics do not favour networks that use the same training-time criterion. We also compare the proposed metrics to human perceptual scores. Generative adversarial networks (GANs) aim to approximate a data distribution P , using a parameterized model distribution Q. They achieve this by jointly optimizing generative and discriminative networks BID9 . GANs are end-to-end differentiable, and samples from the generative network are propagated forward to a discriminative network, and error signals are then propagated backwards from the discriminative network to the generative network. The discriminative network is often viewed as learned, adaptive loss function for the generative network.GANs have achieved state-of-the-art results for a number of applications BID8 , producing more realistic, sharper samples than other popular generative models, such as variational autoencoders BID22 . Because of their success, many GAN frameworks have been proposed. However, it has been difficult to compare these algorithms and understand their strengths and weaknesses because we are currently lacking in quantitative methods for assessing the learned generators.In this work, we propose new metrics for measuring how realistic samples generated from GANs are. These criteria are based on a formulation of divergence between the distributions P and Q BID33 BID38 : DISPLAYFORM0 Here, different choices of µ, υ, and F can correspond to different f -divergences BID33 or different integral probability metrics (IPMs) BID38 . Importantly, J(Q) can be estimated using samples from P and Q, and does not require us to be able to estimate P (x) or Q(x) for samples x. Instead, evaluating J(Q) involves finding the function f ∈ F that is maximally different with respect to P and Q.This measure of divergence between the distributions P and Q is related to the GAN criterion if we restrict the function class F to be neural network functions parameterized by the vector φ and the class of approximating distributions to correspond to neural network generators G θ parameterized by the vector θ, allowing formulation as a min-max problem: H is a Reproducing Kernel Hilbert Space (RKHS) and · L is the Lipschitz constant. For the LS-DCGAN, we used b = 1 and a = 0 BID28 . DISPLAYFORM1 Metric µ υ Function Class GAN (GC) log f − log(1 − f ) X → R + , ∃M ∈ R : |f (x)| ≤ M Least-Squares GAN (LS) −(f − b) DISPLAYFORM2 In this formulation, Q θ corresponds to the generator network's distribution and D φ corresponds to the discriminator network (see BID33 for details).We propose using J(θ) to evaluate the performance of the generator network G θ for various choices of µ and υ, corresponding to different f -divergences or IPMs between distributions P and Q θ , that have been successfully used for GAN training. Our proposed metrics differ from most existing metrics in that they are adaptive, and involve finding the maximum over discriminative networks. We compare four metrics, those corresponding to the original GAN (GC) BID8 , the Least-Squares GAN (LS) BID28 ,the Wasserstein GAN (IW) , and the Maximum Mean Discrepency GAN (MMD) criteria. Choices for µ, υ, and F for these metrics are shown in TAB0 . Our method can easily be extended to other f -divergences or IPMs.To compare these and previous metrics for evaluating GANs, we performed many experiments, training and comparing multiple types of GANs with multiple architectures on multiple data sets. We qualitatively and quantitatively compared these metrics to human perception, and found that our proposed metrics better reflected human perception. We also show that rankings produced using our proposed metrics are consistent across metrics, thus are robust to the exact choices of the functions µ and υ in Equation 2.We used the proposed metrics to quantitatively analyze three different families of GANs: Deep Convolutional Generative Adversarial Networks (DCGAN) BID34 , Least-Squares GANs (LS-DCGAN), and Wasserstein GANs (W-DCGAN), each of which corresponded to a different proposed metric. Interestingly, we found that the different proposed metrics still agreed on the best GAN framework for each dataset. Thus, even though , e.g. for MNIST the W-DCGAN was trained with the IW criterion, LS-DCGAN still outperformed it for the IW criterion.Our analysis also included carrying out a sensitivity analysis with respect to various factors, such as the architecture size, noise dimension, update ratio between discriminator and generator, and number of data points. Our empirical results show that: i) the larger the GAN architecture, the better the results; ii) having a generator network larger than the discriminator network does not yield good results; iii) the best ratio between discriminator and generator updates depend on the data set; and iv) the W-DCGAN and LS-DCGAN performance increases much faster than DCGAN as the number of training examples grows. These metrics thus allow us to tune the hyper-parameters and architectures of GANs based on our proposed method. In this paper, we proposed to use four well-known distance functions as an evaluation metrics, and empirically investigated the DCGAN, W-DCGAN, and LS-DCGAN families under these metrics. Previously, these models were compared based on visual assessment of sample quality and difficulty of training. In our experiments, we showed that there are performance differences in terms of average experiments, but that some are not statistically significant. Moreover, we thoroughly analyzed the performance of GANs under different hyper-parameter settings.There are still several types of GANs that need to be evaluated, such as GRAN BID18 , IW-DCGAN BID12 , BEGAN BID4 , MMDGAN , and CramerGAN (Bellemare et al., 2017) . We hope to evaluate all of these models under this framework and thoroughly analyze them in the future. Moreover, there has been an investigation into taking ensemble approaches to GANs, such as Generative Adversarial Parallelization BID19 . Ensemble approaches have been empirically shown to work well in many domains of research, so it would be interesting to find out whether ensembles can also help in min-max problems. Alternatively, we can also try to evaluate other log-likelihood-based models like NVIL BID30 , VAE BID22 , DVAE BID17 , DRAW BID10 , RBMs BID15 BID35 , NICE Dinh et al. (2014) , etc.Model evaluation is an important and complex topic. Model selection, model design, and even research direction can change depending on the evaluation metric. Thus, we need to continuously explore different metrics and rigorously evaluate new models.
An empirical evaluation on generative adversarial networks
{"src": "scitldr-AIC_train", "paper_id": "SJQHjzZ0-"}