Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> LOSS-VARIANT GANS <s> We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to ½ everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples. <s> BIB001 </s> Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> LOSS-VARIANT GANS <s> The goal of this paper is not to introduce a single algorithm or method, but to make theoretical steps towards fully understanding the training dynamics of generative adversarial networks. In order to substantiate our theoretical analysis, we perform targeted experiments to verify our assumptions, illustrate our claims, and quantify the phenomena. This paper is divided into three sections. The first section introduces the problem at hand. The second section is dedicated to studying and proving rigorously the problems including instability and saturation that arize when training generative adversarial networks. The third section examines a practical and theoretically grounded direction towards solving these problems, while introducing new tools to study them. <s> BIB002 </s> Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> LOSS-VARIANT GANS <s> It is common but challenging to address high-resolution image blending in the automatic photo editing application. In this paper, we would like to focus on solving the problem of high-resolution image blending, where the composite images are provided. We propose a framework called Gaussian-Poisson Generative Adversarial Network (GP-GAN) to leverage the strengths of the classical gradient-based approach and Generative Adversarial Networks. To the best of our knowledge, it's the first work that explores the capability of GANs in high-resolution image blending task. Concretely, we propose Gaussian-Poisson Equation to formulate the high-resolution image blending problem, which is a joint optimization constrained by the gradient and color information. Inspired by the prior works, we obtain gradient information via applying gradient filters. To generate the color information, we propose a Blending GAN to learn the mapping between the composite images and the well-blended ones. Compared to the alternative methods, our approach can deliver high-resolution, realistic images with fewer bleedings and unpleasant artifacts. Experiments confirm that our approach achieves the state-of-the-art performance on Transient Attributes dataset. A user study on Amazon Mechanical Turk finds that the majority of workers are in favor of the proposed method. <s> BIB003
Another design decision in GANs which significantly impacts performance is the choice of loss function in equation BIB001 . While the original GAN work BIB001 has already proved global optimality and the convergence of GANs training. It still highlights the instability problem which can arise when training a GAN. The problem is caused by the global optimality criterion as stated in BIB001 . Global optimality is achieved when an optimal D is reached for any G. So the optimal D is achieved when the derivative of D for the loss in equation (1) equals 0. So we have where x represents the real data and generated data, D * (x) is the optimal discriminator, p r (x) is the real data distribution and p g (x) is the generated data distribution. We have got the optimal discriminator D so far. When we have the optimal D, the loss for G can be visualized by substituting Equation (3) demonstrates the loss function for a GAN when discriminator is optimized and it is related to two important probability measurement metrics. One is Kullback-Leibler (KL) divergence which is defined as and the other is Jensen-Shannon (JS) divergence which is stated as Thus the loss for G regarding the optimal D in equation (3) can be reformulated as which indicates that the loss for G now equally becomes the minimization of the JS divergence between p r and p g . With the training D step by step, the optimization of G will be closer to the minimization of JS divergence between p r and p g . We now start to explain the unstable training problem, where D often easily wins G. This unstable training problem is actually caused by the JS divergence in equation BIB003 . Give an optimal D, the objective of optimization for equation (6) is to move p g toward p r (see Fig. 9 ). JS divergence for the three plots from left to right are 0.693, 0.693 and 0.336, which indicates that JS divergence stays constant (log2=0.693) if there is no overlap between p r and p g . Figure 10 demonstrates the change of JS divergence and its gradient corresponding to the distance between p r and p g . It can be seen that JS divergence is constant and its gradient is almost 0 when the distance is greater than 5, which indicates that training process does not have any effect on G. The gradient of JS divergence for training the G is non-zero only when p g and p r have substantial overlap i.e., the vanishing gradient will arise for G when D is close to optimal. In practice, the possibility that p r and p g do not overlap or have negligible overlap is very high BIB002 . The original GANs work BIB001 also highlights the minimization of −E x∼pg log[D(x)] for training G to avoid a vanishing gradient. However, this training strategy will lead to another problem called mode dropping. First  The alternative loss form for G now can be stated by switching the order of the two sides in equation (7) − where the alternative loss for G in equation (8) is only affected by the first two terms (the last two terms are constant). It can be noticed that the optimization in equation (8) is contradictory because the first term aims to push the generated distribution toward the real distribution while the second term aim to push in the opposite direction (the negative sign). This will cause an unstable numerical gradient for training G. More importantly, KL divergence is an asymmetrical distribution measurement highlighted below The penalization for two instances of poor performance made by G are totally different. The first instance of poor performance is that G is not producing a reasonable range of samples and yet incurs a very small penalization. The second instance of poor performance concerns G producing implausible samples but has very large penalization. The first example concerns the fact that the generated samples lack diversity while the second concerns that fact that the generated samples are not accurate. Considering this first case, G generates repeated but "safe" samples instead of taking risk to generate diverse but "unsafe" samples, which leads to the mode collapse problem. In summary, using the original loss in equation (1) will result in the vanishing gradient for training G and using the alternative loss in equation (8) will incur the mode collapse problem. These kind of problems cannot be solved by changing the GANs architectures. Therefore, it could be argued that ultimate GANs problem stems from the design of the loss function and that innovative ideas for this redesign of the loss function may solve the problem. Loss-variant GANs have been researched extensively to improve the stability of training GANs.