Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> Summary <s> The explosion of image data on the Internet has the potential to foster more sophisticated and robust models and algorithms to index, retrieve, organize and interact with images and multimedia data. But exactly how such data can be harnessed and organized remains a critical problem. We introduce here a new database called “ImageNet”, a large-scale ontology of images built upon the backbone of the WordNet structure. ImageNet aims to populate the majority of the 80,000 synsets of WordNet with an average of 500-1000 clean and full resolution images. This will result in tens of millions of annotated images organized by the semantic hierarchy of WordNet. This paper offers a detailed analysis of ImageNet in its current state: 12 subtrees with 5247 synsets and 3.2 million images in total. We show that ImageNet is much larger in scale and diversity and much more accurate than the current image datasets. Constructing such a large-scale database is a challenging task. We describe the data collection scheme with Amazon Mechanical Turk. Lastly, we illustrate the usefulness of ImageNet through three simple applications in object recognition, image classification and automatic object clustering. We hope that the scale, accuracy, diversity and hierarchical structure of ImageNet can offer unparalleled opportunities to researchers in the computer vision community and beyond. <s> BIB001 </s> Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> Summary <s> Synthesizing high resolution photorealistic images has been a long-standing challenge in machine learning. In this paper we introduce new methods for the improved training of generative adversarial networks (GANs) for image synthesis. We construct a variant of GANs employing label conditioning that results in 128x128 resolution image samples exhibiting global coherence. We expand on previous work for image quality assessment to provide two new analyses for assessing the discriminability and diversity of samples from class-conditional image synthesis models. These analyses demonstrate that high resolution samples provide class information not present in low resolution samples. Across 1000 ImageNet classes, 128x128 samples are more than twice as discriminable as artificially resized 32x32 samples. In addition, 84.7% of the classes have samples exhibiting diversity comparable to real ImageNet data. <s> BIB002 </s> Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> Summary <s> We describe a new training methodology for generative adversarial networks. The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses. This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2. We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10. Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator. Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation. As an additional contribution, we construct a higher-quality version of the CelebA dataset. <s> BIB003 </s> Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> Summary <s> Unsupervised learning with generative adversarial networks (GANs) has proven hugely successful. Regular GANs hypothesize the discriminator as a classifier with the sigmoid cross entropy loss function. However, we found that this loss function may lead to the vanishing gradients problem during the learning process. To overcome such a problem, we propose in this paper the Least Squares Generative Adversarial Networks (LSGANs) which adopt the least squares loss function for the discriminator. We show that minimizing the objective function of LSGAN yields minimizing the Pearson X2 divergence. There are two benefits of LSGANs over regular GANs. First, LSGANs are able to generate higher quality images than regular GANs. Second, LSGANs perform more stable during the learning process. We evaluate LSGANs on LSUN and CIFAR-10 datasets and the experimental results show that the images generated by LSGANs are of better quality than the ones generated by regular GANs. We also conduct two comparison experiments between LSGANs and regular GANs to illustrate the stability of LSGANs. <s> BIB004 </s> Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> Summary <s> We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spectral norm regularization, which penalizes the high spectral norm of weight matrices in neural networks. We provide supportive evidence for the abovementioned hypothesis by experimentally confirming that the models trained using spectral norm regularization exhibit better generalizability than other baseline methods. <s> BIB005 </s> Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> Summary <s> Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal. To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specific to such scale. We find that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick," allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input. Our modifications lead to models which set the new state of the art in class-conditional image synthesis. When trained on ImageNet at 128x128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.5 and Frechet Inception Distance (FID) of 7.4, improving over the previous best IS of 52.52 and FID of 18.6. <s> BIB006
We have provided an overview of architecture-variant GANs which aim to improve performance based on the three key challenges: (1 variants are able to improve image quality. SAGAN is proposed for improving the capacity of multi-class learning in GANs, the goal of which is to produce more diverse images. Benefiting from the SAGAN architecture, BigGAN is designed for improving both image quality and image diversity. It should be noted that both PROGAN and BigGAN are able to produce high resolution images. BigGAN realizes this higher resolution by increasing the batch size and the authors mention that a progressive growing BIB003 operation is unnecessary when the batch size is large enough (2048 used in the original paper BIB006 ). However, a progressive growing operation is still needed when GPU memory is limited (a large batch size is hungry for GPU memory). Benefiting from spectrum normalization (SN), which will be discussed in loss-variant GANs part, both SAGAN and BigGAN is effective for the vanishing gradient challenge. These milestone architecture-variants indicate a strong advantage of GANs -compatibility, where a GAN is open to any type of neural architecture. This property enables GANs to be applied to many different applications. Regarding the improvements achieved by different architecture-variant GANs, we next present an analysis on the interconnections and comparisons between the architecture-variants presented here. Starting with the FCGAN described in the original GAN literature, this architecture-variant can only generate simple image datasets. Such a limitation is caused by the network architecture, where the capacity of FC networks is very limited. Research on improving the performance of GANs starts from designing more complex architectures for GANs. A more complex image datasets (e.g., ImageNet) has higher resolution and diversity comparing to simple image datasets (e.g., MNIST) and needs accordingly more sophisticated approaches. In the context of producing higher resolution images, one obvious approach is to increase the size of generator. LAPGAN and DCGAN up-sample the generator based on such a perspective. Benefiting from the concise deconvolutional up-sampling process and easy generalization of DCGAN, the architecture in DCGAN is more widely used in the GANs literature. It should be noticed that most GANs in the computer vision area use the deconvolutional neural network as the generator, which is first used in DCGAN. Therefore, DCGAN is one of the classical GAN-variants in the literature. The ability to produce high quality images is an important aspect of GANs clearly. This can be improved through judicious choice of architecture. BEGAN and PRO-GAN demonstrate approaches from this perspective. With the same architecture used for the generator in DCGAN, BEGAN redesigns the discriminator by including encoder and decoder, where the discriminator tries to distinguish the difference between the generated and autoencoded images in pixel space. Image quality has been improved in this case. Based on DCGAN, PROGAN demonstrates a progressive approach that incrementally trains an architecture similar to DCGAN. This novel approach cannot only improve image quality but also produce higher resolution images. Producing diverse images is the most challenging task for GANs and it is very difficult for GANs to successfully produce images such as those represented in the ImageNet sets. It is difficult for traditional CNNs to learn global and long-range dependencies from images. Thanks to selfattention mechanism though, approaches such as those in SAGAN integrate self-mechanisms to both discriminator and generator, which helps GANs a lot in terms of learning multi-class images. Moreover, BigGAN, which can be considered an extension of SAGAN, introduces a deeper GAN architecture with a very large batch size, which produces high quality and diverse images as in ImageNet and is the current state-of-the-art. We explain the training problems (mode collapse and vanishing gradient for G) in the original GAN and we have introduced loss-variant GANs in the literature, which are mainly proposed for improving the performance of GANs in terms of three key aspects. Figure 17 efficacy of loss-variant GANs for the challenges. Losses of LSGAN, RGAN and WGAN are very similar to the original GAN loss. We use a toy example (i.e., two distributions used in Fig. 9 ) to demonstrate the G loss regarding the distance between real data distribution and generated data distribution in Fig. 18 . It can be seen that RGAN and WGAN are able to inherently solve the vanishing gradient problems for the generator when the discriminator is optimized. LSGAN on the contrary still suffers from a vanishing gradient for the generator, however, it is able to provide a better gradient compared to the original GAN in Fig. 10 when the distance between the real data distribution and the generated data distribution is relatively small. This is demonstrated in the original paper BIB004 where LSGAN is shown to be better easier to push generated samples to the boundary made by discriminator. Table 1 gives details of the properties of each lossvariant GAN. WGAN, LSGAN, LS-GAN, RGAN and SN-GAN are proposed to overcome the vanishing gradient for G. LSGAN argues that the vanishing gradient is mainly caused by the sigmoid function in the discriminator so it uses a least squares loss to optimize the GAN. LSGAN turns out to be the optimization on Pearson χ 2 divergence and remedies the vanishing gradient problem. WGAN uses Wasserstein (or Earth mover) distance as the loss. Compared to JS divergence, Wasserstein distance is smoother and there is no sudden change with respect to the distance between real samples and generated samples. To be able to use Wasserstein distance as the loss, the discriminator must be Lipschitz continuous, where WGAN deploys the parameter clipping to force discriminator satisfy the Lipschitz continuity. However, it causes problems such as most of parameters in the discriminator locates to the edges of clipping range, which leads to the low capacity of discriminator. WGAN-GP is proposed the use of gradient penalty to make discriminator is Lipschitz continuous, which successfully solves the problems in WGAN. LS-GAN proposes to use a margin that is enforced to separate real samples from generated samples, which restricts the modelling capability of discriminator. It solves the vanishing gradient problem for the generator because this problem arises when the discriminator is optimized. RGAN is a unified framework that is suitable for all IPM-based GANs e.g., WGAN. RGAN adds discriminant information to GANs for better learning. SN-GAN proposes an elegant way for optimizing a GAN. As mentioned before, a Lipschitz continuous discriminator is important for stable learning, vanishing gradient and so on. SN-GAN proposes spectral normalization BIB005 to constrain the discriminator under the Lipschitz continuous requirement. SN-GAN is the first GAN (we do not consider AC-GANs BIB002 because an ensemble of 100 AC-GANs is used for ImageNet datasets BIB001 .) that has been successfully applied to ImageNet datasets. In theory, spectral normalization as demonstrated in the SN-GAN could be applied to every GAN type. SAGAN and BigGAN BIB006 ,  both deploy spectral normalization and achieve good results with ImageNet. Loss-variant GANs are able to be applied to architecturevariants. However, SN-GAN and RGAN show stronger generalization abilities compared to other loss-variants, where these two loss-variants can be deployed by other types of loss-variants. Spectral normalization can be applied to any GAN-variant while the RGAN concept can be applied to any IPM-based GAN. We strongly recommend the use of spectral normalization for all GANs applications as described here. There are a number of loss-variant GANs mentioned in this paper which is able to solve the mode collapse and unstable training problem. Details are given in Table 1 .