Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant amount of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of ``tricks". The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, and neural architectures. In this work we take a sober view of the current state of GANs from a practical perspective. We reproduce the current state of the art and go beyond fairly exploring the GAN landscape. We discuss common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub. Deep generative models are a powerful class of unsupervised machine learning models. The power of these models was recently harnessed in a variety of applications, including image generation, learned compression, and domain transfer BID13 Radford et al., 2016; BID0 BID0 . Generative adversarial networks BID8 are one of the main approaches to learning such models in a fully unsupervised fashion. The GAN framework can be viewed as a two-player game where the first player, the generator, is learning to transform some simple input distribution (usually a standard multivariate Normal or uniform) to a distribution on the space of images, such that the second player, the discriminator, cannot tell whether the samples belong to the true distribution or were synthesized. Both players aim to minimize their own loss and the solution to the game is the Nash equilibrium where neither player can improve their loss unilaterally. This powerful framework can also be derived by minimizing a divergence between the model distribution and the true distribution BID20 .Training GANs involves solving a minimax problem over the parameters of the generator and the discriminator which are usually parameterized as deep convolutional neural networks. Consequently , this minimax problem is notoriously hard to solve in practice. As a result , a plethora of loss functions, regularization and normalization schemes, coupled with neural architecture choices, have been proposed BID8 Salimans et al., 2016; BID19 BID9 BID18 .Our contributions . In this work we provide a thorough empirical analysis of these competing approaches, and help the researchers and practitioners navigate this space. We first define the GAN landscape -the set of loss functions, normalization and regularization schemes, and the most commonly used architectures. We explore this search space on several modern large-scale data sets by means of hyperparameter optimization, considering both "good" sets of hyperparameters reported in the literature, as well as ones obtained by Gaussian Process regression. By analyzing the impact of the loss function, we conclude that the non-saturating loss is sufficiently stable across data sets, architectures and hyperparameters. We then proceed to decompose the effect of various normalization and regularization schemes, as well as varying architectures. We show that both gradient penalty BID9 as well as spectral normalization BID19 are useful in the context of high-capacity architectures. Finally, we discuss some common pitfalls, reproducibility issues, and practical considerations. We provide reference implementations, including training and evaluation code on Github 1 and provide pre-trained models on TensorFlow Hub. 2 2 THE GAN LANDSCAPE Given that there are 4 major components (loss, architecture, regularization, normalization) to analyze for each data set, it is infeasible to explore the whole landscape. Hence, we opt for a more pragmatic solution -we keep some dimensions fixed, and vary the others. For each experiment we highlight three aspects: (1) FID distribution of the top 5% of the trained models, (2) the corresponding sample diversity score, and (3) the tradeoff between the computational budget (i.e. number of models to train) and model quality in terms of FID. Each model was retrained 5 times with a different random seed and we report the median score. The variance for models obtained by Gaussian Process regression is handled implicitly so we train each model once. In this work we study the GAN landscape: losses, regularization and normalization schemes, and neural architectures, and their impact on the on the quality of generated samples which we assess by recently introduced quantitative metrics. Our fair and thorough empirical evaluation suggests that one should consider non-saturating GAN loss and spectral normalization as default choices when applying GANs to a new data set. Given additional computational budget, we suggest adding the gradient penalty from BID9 and train the model until convergence. Furthermore, additional marginal gains can be obtained by combining normalization and regularization empirically confirming the importance of the Lipschitz constant of the discriminator. Furthermore, both types of architectures proposed up-to this point perform reasonably well. A separate ablation study uncovered that most of the tricks applied in the ResNet style architectures lead to marginal changes in the quality and should be avoided due to the high computational cost. As a result of this large-scale study we identify the common pitfalls standing in the way of accurate and fair comparison and propose concrete actions to demystify the future results -issues with metrics, data set preprocessing, non-determinism, and missing implementation details are particularly striking. We hope that this work, together with the open-sourced reference implementations and trained models, will serve as a solid baseline for future GAN research. We present an empirical study with SNDCGAN and ResNet CIFAR architectures on CIFAR10 in figure 5 and figure 6 . In addition to Section 3.1, we evaluate one more kind of loss on CIFAR10. Here HG, NS and WGAN stand for hinge loss, non saturating loss and Wasserstein loss respectively. We observe that hinge loss performs very similar to non-saturating loss. DISPLAYFORM0
A sober view on the current state of GANs from a practical perspective
{"src": "scitldr-AIC_train", "paper_id": "rkGG6s0qKQ"}