Improved generative adversarial network (Improved GAN) is a successful method of using generative adversarial models to solve the problem of semi-supervised learning. However, it suffers from the problem of unstable training. In this paper, we found that the instability is mostly due to the vanishing gradients on the generator. To remedy this issue, we propose a new method to use collaborative training to improve the stability of semi-supervised GAN with the combination of Wasserstein GAN. The experiments have shown that our proposed method is more stable than the original Improved GAN and achieves comparable classification accuracy on different data sets. Generative adversarial networks (GANs) BID3 have been recently studied intensively and achieved great success in deep learning domain BID14 BID9 BID15 . A typical GAN simulates a two-player minimax game, where one aims to fool the other and the overall system is finally able to achieve equilibrium.Specifically speaking, we have a generator G to generate fake data G(z) from a random variable z whose distribution density is p(z), and also we have a discriminator D(x) to discriminate the real x from the generated data G(z), where x ∼ p r (x) and p r is the distribution density of real data. We optimize the two players G(z) and D(x) by solving the following minimax problem: DISPLAYFORM0 This method is so called as the original GAN BID3 . After this, many different types of GANs have been proposed, e.g., least-squared GAN BID9 , cat-GAN BID15 , W-GAN , Improved GAN BID14 , so on and so forth, focusing on improving the performance of GANs and extending the GAN idea to other application scenarios.For instance, the original GAN is trained in a completely unsupervised learning way BID3 , along with many variants, such as LS-GAN and cat-GAN. It was later extended to semi-supervised learning. In BID14 , Salimans et al. proposed the Improved GAN to enable generation and classification of data simultaneously. In BID7 , Li et al. extended this method to consider conditional data generation.Another issue regarding the unsupervised learning of GANs is the lack of training stability in the original GANs, mostly because of dimension mismatch . A lot of efforts have been dedicated to solve this issue. For instance, in , the authors theoretically found that the instability problem and dimension mismatch of the unsupervised learning GAN was due to the maxing out of Jensen-Shannon divergence between the true and fake distribution and therefore proposed using the Wasserstein distance to train GAN. However, to calculate the Wasserstein distance, the network functions are required to be 1-Lipschitz, which was simply implemented by clipping the weights of the networks in . Later, Gulrajani et. al. improved it by using gradient penalty BID4 . Besides them, the same issue was also addressed from different perspectives. In BID13 , Roth et al. used gradient norm-based regularization to smooth the f-divergence objective function so as to reduce dimension mismatch. However, the method could not directly work on f-divergence, which was intractable to solve, but they instead optimized its variational lower bound. Its converging rate is still an open question and its computational complexity may be high. On the other hand, there were also some efforts to solve the issue of mode collapse, so as to try to stabilize the training of GANs from another perspective, including the unrolled method in BID10 , mode regularization with VAEGAN (Che et al., 2016) , and variance regularization with bi-modal Gaussian distributions BID5 . However, all these methods were investigated in the context of unsupervised learning. Instability issue for semi-supervised GAN is still open.In this work, we focus on investigating the training stability issue for semi-supervised GAN. To the authors' best knowledge, it is the first work to investigate the training instability for semi-supervised GANs, though some were done for unsupervised GANs as aforementioned. The instability issue of the semi-supervised GAN BID14 is first identified and analyzed from a theoretical perspective. We prove that this issue is in fact caused by the vanishing gradients theorem on the generator. We thus propose to solve this issue by using collaborative training to improve its training stability. We theoretically show that the proposed method does not have vanishing gradients on the generator, such that its training stability is improved. Besides the theoretical contribution, we also show by experiments that the proposed method can indeed improve the training stability of the Improved GAN, and at the same time achieve comparable classification accuracy.It is also worth to note that BID7 proposed the Triple GAN that also possessed two discriminators. However, its purpose is focused on using conditional probability training (the original GAN uses unconditional probability) based on data labels to improve the training of GAN, but not on solving the instability issue. Therefore, the question of instability for the Triple GAN is still unclear. More importantly, the method, collaborative training, proposed for exploring the data labels with only unconditional probability in this paper , can also be applied to the Triple GAN to improve its training stability, in the case of conditional probability case.The rest of the paper is organized as follows: in Section 2, we present the generator vanishing gradient theorem of the Improved GAN. In Section 3, we propose a new method, collaborative training Wasserstein GAN (CTW-GAN) and prove its nonvanishing gradient theorem. In Section 4, we present our experimental results and finally give our conclusion in Section 5. In the paper, we study the training instability issue of semi-supervised improved GAN. We have found that the training instability is mainly due to the vanishing gradients on the generator of the Improved GAN. In order to make the training of the Improved GAN more stable, we propose a collaborative training method to combine Wasserstein GAN with the semi-supervised improved GAN. Both theoretical analysis and experimental results on MNIST and CIFAR-10 have shown the effectiveness of the proposed method to improve training stability of the Improved GAN. In addition, it also achieves the classification accuracy comparable to the original Improved GAN.
Improve Training Stability of Semi-supervised Generative Adversarial Networks with Collaborative Training
{"src": "scitldr-AIC_train", "paper_id": "ry4SNTe0-"}