Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> Spectral normalization GAN (SN-GAN) <s> This paper contains basic results that are useful for building algorithms for the optimization of Lipschitz continuous functionsf on compact subsets of En. In this settingf is differentiable a.e. The theory involves a set-valued mappingxźźźf(x) whose range is the convex hull of existing values of źf and limits of źf on a closedź-ball,B(x, ź). ::: ::: As an application, simple descent algorithms are formulated that generate sequence {xk} whose distance from some stationary set (see Section 2) is 0, and where {f(xk)} decreases monotonously. This is done with the aid of anyone of the following three hypotheses: ::: ::: Forź arbitrarily small, a point is available that in arbitrarily close to:(1)the minimizer off onB(x, ź),(2)the closest point inźźf(x) to the origin,(3)ź(h) ź źźf(x), where [ź(h), h] = max {[ź, h]: ź ź źźf(x)}. ::: ::: Observe that these three problems are simplified iff has a tractable local approximation. ::: ::: The minimax problem is taken as an example, and algorithms for it are sketched. For this example, all three hypotheses may be satisfied. A class of functions called uniformly-locally-convex is introduced that is also tractable. <s> BIB001 </s> Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> Spectral normalization GAN (SN-GAN) <s> Ordinary differential and functional-differential inclusions with compact right-hand sides are considered. Stability theorems of Filippov's type in the convex and nonconvex case are proved under a one-sided Lipschitz condition, which extends the notions of Lipschitz continuity, dissipativity, and the uniform one-sided Lipschitz condition for set-valued mappings. The accuracy of approximation of the solution sets by means of the Euler discretization scheme for both types of inclusions is estimated. <s> BIB002 </s> Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> Spectral normalization GAN (SN-GAN) <s> Generative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only low-quality samples or fail to converge. We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior. We propose an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input. Our proposed method performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets and language models over discrete data. We also achieve high quality generations on CIFAR-10 and LSUN bedrooms. <s> BIB003 </s> Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> Spectral normalization GAN (SN-GAN) <s> In this paper, we present the Lipschitz regularization theory and algorithms for a novel Loss-Sensitive Generative Adversarial Network (LS-GAN). Specifically, it trains a loss function to distinguish between real and fake samples by designated margins, while learning a generator alternately to produce realistic samples by minimizing their losses. The LS-GAN further regularizes its loss function with a Lipschitz regularity condition on the density of real data, yielding a regularized model that can better generalize to produce new data from a reasonable number of training examples than the classic GAN. We will further present a Generalized LS-GAN (GLS-GAN) and show it contains a large family of regularized GAN models, including both LS-GAN and Wasserstein GAN, as its special cases. Compared with the other GAN models, we will conduct experiments to show both LS-GAN and GLS-GAN exhibit competitive ability in generating new images in terms of the Minimum Reconstruction Error (MRE) assessed on a separate test set. We further extend the LS-GAN to a conditional form for supervised and semi-supervised learning problems, and demonstrate its outstanding performance on image classification tasks. <s> BIB004
SN-GAN proposes the use of weight normalization to stabilize the training of the discriminator. This technique is computationally light and easily applied to existing GANs. Previous work for stabilizing the training of GANs BIB003 , [88] , BIB004 emphasizes the importance that D should be from the set of K-Lipshitz continuous functions. Popularly speaking, Lipschitz continuity BIB002 - BIB001 is more strict than the continuity, which describes that the function does not change rapidly. This smooth D is of benefit in stabilizing the training of GANs. The work mentioned previously focused on the control of the Lipschitz constant of the discriminator function. This work demonstrates an alternative simpler way to control the Lipschitz constant through spectral normalization of each layer for D. Spectral normalization is performed asW where W represents weights on each layer for D and σ(W) is the L 2 matrix norm of W. The paper proves this will make f ≤ 1. The fast approximation for the σ(W) is also demonstrated in the original paper.