We study the problem of alleviating the instability issue in the GAN training procedure via new architecture design. The discrepancy between the minimax and maximin objective values could serve as a proxy for the difficulties that the alternating gradient descent encounters in the optimization of GANs. In this work, we give new results on the benefits of multi-generator architecture of GANs. We show that the minimax gap shrinks to \epsilon as the number of generators increases with rate O(1/\epsilon). This improves over the best-known result of O(1/\epsilon^2). At the core of our techniques is a novel application of Shapley-Folkman lemma to the generic minimax problem, where in the literature the technique was only known to work when the objective function is restricted to the Lagrangian function of a constraint optimization problem. Our proposed Stackelberg GAN performs well experimentally in both synthetic and real-world datasets, improving Frechet Inception Distance by 14.61% over the previous multi-generator GANs on the benchmark datasets. Generative Adversarial Nets (GANs) are emerging objects of study in machine learning, computer vision, natural language processing, and many other domains. In machine learning, study of such a framework has led to significant advances in adversarial defenses BID25 BID22 and machine security BID3 BID22 . In computer vision and natural language processing, GANs have resulted in improved performance over standard generative models for images and texts BID11 , such as variational autoencoder BID14 and deep Boltzmann machine BID20 . A main technique to achieve this goal is to play a minimax two-player game between generator and discriminator under the design that the generator tries to confuse the discriminator with its generated contents and the discriminator tries to distinguish real images/texts from what the generator creates. Despite a large amount of variants of GANs, many fundamental questions remain unresolved. One of the long-standing challenges is designing universal, easy-to-implement architectures that alleviate the instability issue of GANs training. Ideally, GANs are supposed to solve the minimax optimization problem BID11 , but in practice alternating gradient descent methods do not clearly privilege minimax over maximin or vice versa (page 35, Goodfellow (2016) ), which may lead to instability in training if there exists a large discrepancy between the minimax and maximin objective values. The focus of this work is on improving the stability of such minimax game in the training process of GANs. To alleviate the issues caused by the large minimax gap, our study is motivated by the so-called Stackelberg competition in the domain of game theory. In the Stackelberg leadership model, the players of this game are one leader and multiple followers, where the leader firm moves first and then the follower firms move sequentially. It is known that the Stackelberg model can be solved to find a subgame perfect Nash equilibrium. We apply this idea of Stackelberg leadership model to the architecture design of GANs. That is, we design an improved GAN architecture with multiple generators (followers) which team up to play against the discriminator (leader). We therefore name our model Stackelberg GAN. Our theoretical and experimental results establish that: GANs with multi-generator architecture have smaller minimax gap, and enjoy more stable training performances. Our Contributions. This paper tackles the problem of instability during the GAN training procedure with both theoretical and experimental results. We study this problem by new architecture design. Figure 1: Stackelberg GAN stabilizes the training procedure on a toy 2D mixture of 8 Gaussians. Top Row: Standard GAN training. It shows that several modes are dropped. Bottom Row: Stackelberg GAN training with 8 generator ensembles, each of which is denoted by one color. We can see that each generator exactly learns one mode of the distribution without any mode being dropped.Under review as a conference paper at ICLR 2019 (a) Step 0 Standard GAN training. It shows that several modes are dropped. Bottom Row: Stackelberg GAN training with 8 generator ensembles, each of which is denoted by one color. We can see that each generator exactly learns one mode of the distribution without any mode being dropped.• We propose Stackelberg GAN framework of having multiple generators in the GAN architecture. Our framework is general that can be applied to all variants of GANs, e.g., vanilla GAN, Wasserstein GAN, etc. It is built upon the idea of jointly optimizing an ensemble of GAN losses w.r.t. all pairs of discriminator and generator. Differences with prior work. Although the idea of having multiple generators in the GAN architecture is not totally new, e.g., MIX+GAN BID1 and MGAN BID13 , there are key differences between Stackelberg GAN and prior work. a) In MGAN BID13 , various generators are combined as a mixture of probabilistic models with assumption that the generators and discriminator have enough capacity. In contrast, in the Stackelberg GAN model we uniformly ensemble the losses of various standard GAN without any assumption on the model capacity. b) In MIX+GAN BID1 , the losses are ensembled with learned weights and an extra regularization term, which discourages the weights being too far away from uniform. We find it slightly unnecessary because the expressive power of each generator already allows implicit scaling of each generator. To the contrary, in the Stackelberg GAN we apply equal weights for all generators.• We prove that the minimax duality gap shrinks as the number of generators increases (see Theorem 1 and Corollary 2). Unlike the previous work, our result has no assumption on the expressive power of generators and discriminator, but instead depends on their non-convexity. With extra condition on the expressive power of generators, we show that Stackelberg GAN is able to achieve ✏-approximate equilibrium with e O(1/✏) generators (see Theorem 3). This Stackelberg GAN training with 10 generator ensembles on real images without cherry pick, where each row corresponds to one generator. We can see that each generator exactly learns one mode of the distribution without any mode being dropped.[Pengtao: It is kind of abrupt that you say "Stackelberg GAN stabilizes the training procedure" in the beginning sentence, then the rest talks about losing mode. In the introduction, a convincing tie between instability and mode collapse is still missing.]• We propose Stackelberg GAN framework of having multiple generators in the GAN architecture. Our framework is general that can be applied to all variants of GANs, e.g., vanilla GAN, Wasserstein GAN, etc. It is built upon the idea of jointly optimizing an ensemble of GAN losses w.r.t. all pairs of discriminator and generator. Differences with prior work . Although the idea of having multiple generators in the GAN architecture is not totally new, e.g., MIX+GAN BID1 and MGAN BID13 , there are key differences between Stackelberg GAN and prior work. a) In MGAN BID13 , various generators are combined as a mixture of probabilistic models with assumption that the generators and discriminator have enough capacity. In contrast, in the Stackelberg GAN model we uniformly ensemble the losses of various standard GAN without any assumption on the model capacity. b) In MIX+GAN BID1 , the losses are ensembled with learned weights and an extra regularization term, which discourages the weights being too far away from uniform. We find it slightly unnecessary because the expressive power of each generator already allows implicit scaling of each generator. To the contrary, in the Stackelberg GAN we apply equal weights for all generators.• We prove that the minimax duality gap shrinks as the number of generators increases (see Theorem 1 and Corollary 2). Unlike the previous work, our result has no assumption on the • We propose the Stackelberg GAN framework of multiple generators in the GAN architecture. Our framework is general since it can be applied to all variants of GANs, e.g., vanilla GAN, Wasserstein GAN, etc. It is built upon the idea of jointly optimizing an ensemble of GAN losses w.r.t. all pairs of discriminator and generator. Differences from prior work. Although the idea of having multiple generators in the GAN architecture is not totally new, e.g., MIX+GAN BID1 , MGAN BID13 , MAD-GAN BID9 and GMAN BID8 , there are key differences between Stackelberg GAN and prior work. a) In MGAN BID13 and MAD-GAN BID9 , various generators are combined as a mixture of probabilistic models with assumption that the generators and discriminator have infinite capacity. Also, they require that the generators share common network parameters. In contrast, in the Stackelberg GAN model we allow various sampling schemes beyond the mixture model, e.g., each generator samples a fixed but unequal number of data points independently. Furthermore, each generator has free parameters . We also make no assumption on the model capacity in our analysis. This is an important research question as raised by BID2 . b) In MIX+GAN BID1 , the losses are ensembled with learned weights and an extra regularization term, which discourages the weights being too far away from uniform. We find it slightly unnecessary because the expressive power of each generator already allows implicit scaling of each generator. In the Stackelberg GAN, we apply equal weights for all generators and obtain improved guarantees. c) In GMAN BID8 , there are multiple discriminators while it is unclear in theory why multi-discriminator architecture works well. In this paper, we provide formal guarantees for our model . • We prove that the minimax duality gap shrinks as the number of generators increases (see Theorem 1 and Corollary 2). Unlike the previous work, our result has no assumption on the expressive power of generators and discriminator, but instead depends on their non-convexity. With extra condition on the expressive power of generators, we show that Stackelberg GAN is able to achieve -approximate equilibrium with O(1/ ) generators (see Theorem 3). This improves over the best-known result in BID1 which requires generators as many as O(1/ 2 ). At the core of our techniques is a novel application of the ShapleyFolkman lemma to the generic minimax problem, where in the literature the technique was only known to work when the objective function is restricted to the Lagrangian function of a constrained optimization problem . This results in tighter bounds than that of the covering number argument as in BID1 . We also note that MIX+GAN is a heuristic model which does not exactly match the theoretical analysis in BID1 , while this paper provides formal guarantees for the exact model of Stackelberg GAN.• We empirically study the performance of Stackelberg GAN for various synthetic and real datasets. We observe that without any human assignment, surprisingly, each generator automatically learns balanced number of modes without any mode being dropped (see FIG2 ). Compared with other multi-generator GANs with the same network capacity, our experiments show that Stackelberg GAN enjoys 26.76 Fréchet Inception Distance on CIFAR-10 dataset while prior results achieve 31.34 (smaller is better), achieving an improvement of 14.61%. In this work, we tackle the problem of instability during GAN training procedure, which is caused by the huge gap between minimax and maximin objective values. The core of our techniques is a multi-generator architecture. We show that the minimax gap shrinks to as the number of generators increases with rate O(1/ ), when the maximization problem w.r.t. the discriminator is concave. This improves over the best-known results of O(1/ 2 ). Experiments verify the effectiveness of our proposed methods. TAB5 is by the weak duality. Thus it suffices to prove the other side of the inequality. All notations in this section are defined in Section 3.1. We first show that DISPLAYFORM0 Denote by DISPLAYFORM1 We have the following lemma.Lemma 4. We have DISPLAYFORM2 Proof. By the definition of p(0), we have p(0) = inf γ1,...,γ I ∈R g sup θ∈R t Φ(γ 1 , ..., γ I ; θ). Since (clp)(·) is the convex closure of function p(·) (a.k.a. weak duality theorem), we have (clp)(0) ≤ p(0). We now show that sup DISPLAYFORM3 Note that p(u) = inf γ1,...,γ I ∈R g p γ1,...,γ I (u), where p γ1,...,γ I (u) = sup θ∈R t { Φ(γ 1 , ..., γ I ; θ) − u T θ} = (− Φ(γ 1 , ..., γ I ; ·)) * (−u), and that . We have the following lemma. DISPLAYFORM4 Lemma 5. Under the assumption in Theorem 1, DISPLAYFORM5 Proof. We note that DISPLAYFORM6 where u 1 , ..., u I , u ∈ R t . Therefore, DISPLAYFORM7 Consider the subset of R t+1 : DISPLAYFORM8 Define the vector summation DISPLAYFORM9 is continuous and domh i is compact, the set DISPLAYFORM10 DISPLAYFORM11 We apply Lemma 6 to prove Lemma 5 with m = t + 1. Let (r, w) ∈ conv(Y) be such that r = 0, and w =clp(0). DISPLAYFORM12 i ∈I DISPLAYFORM13 Representing elements of the convex hull of DISPLAYFORM14 by Carathéodory theorem, we have that for each i ∈ I, there are vectors {u DISPLAYFORM15 Recall that we definȇ DISPLAYFORM16 and DISPLAYFORM17 We have for i ∈ I, DISPLAYFORM18 Thus, by Eqns. FORMULA27 and FORMULA30 , we have DISPLAYFORM19 Therefore, we have DISPLAYFORM20 (by Eqns. FORMULA28 and FORMULA33 ) DISPLAYFORM21 , (by Lemma 6) as desired.By Lemmas 4 and 5, we have proved that DISPLAYFORM22 To prove Theorem 1, we note that DISPLAYFORM23 When φ(γ i ; θ) is concave and closed w.r.t. discriminator parameter θ, we have clφ = φ. Thus, ∆ minimax θ = ∆ maximin θ = 0 and 0 ≤ w * − q * ≤ .
We study the problem of alleviating the instability issue in the GAN training procedure via new architecture design, with theoretical guarantees.
{"src": "scitldr-AIC_train", "paper_id": "SJxCsj0qYX"}