In the field of Generative Adversarial Networks (GANs), how to design a stable training strategy remains an open problem. Wasserstein GANs have largely promoted the stability over the original GANs by introducing Wasserstein distance, but still remain unstable and are prone to a variety of failure modes. In this paper, we present a general framework named Wasserstein-Bounded GAN (WBGAN), which improves a large family of WGAN-based approaches by simply adding an upper-bound constraint to the Wasserstein term. Furthermore, we show that WBGAN can reasonably measure the difference of distributions which almost have no intersection. Experiments demonstrate that WBGAN can stabilize as well as accelerate convergence in the training processes of a series of WGAN-based variants. Over the past few years, Generative Adversarial Networks (GANs) have shown impressive results in many generative tasks. They are inspired by the game theory, that two models compete with each other: a generator which seeks to produce samples from the same distribution as the data, and a discriminator whose job is to distinguish between real and generated data. Both models are forced stronger simultaneously during the training process. GANs are capable of producing plausible synthetic data across a wide diversity of data modalities, including natural images (Karras et al., 2017; Brock et al., 2018; Lucic et al., 2019) , natural language (Press et al., 2017; Lin et al., 2017; Rajeswar et al., 2017) , music Mogren, 2016; Dong et al., 2017; Dong & Yang, 2018) , etc. Despite their success, it is often difficult to train a GAN model in a fast and stable way, and researchers are facing issues like vanishing gradients, training instability, mode collapse, etc. This has led to a proliferation of works that focus on improving the quality of GANs by stabilizing the training procedure (Radford et al., 2015; Salimans et al., 2016; Zhao et al., 2016; Nowozin et al., 2016; Qi, 2017; Deshpande et al., 2018) . In particular, introduced a variant of GANs based on the Wasserstein distance, and releases the problem of gradient disappearance to some extent. However, WGANs limit the weight within a range to enforce the continuity of Lipschitz, which can easily cause over-simplified critic functions (Gulrajani et al., 2017) . To solve this issue, Gulrajani et al. (2017) proposed a gradient penalty method termed WGAN-GP, which replaces the weight clipping in WGANs with a gradient penalty term. As such, WGAN-GP provides a more stable training procedure and succeeds in a variety of generating tasks. Based on WGAN-GP, more works (Wei et al., 2018; Petzka et al., 2017; Wu et al., 2018; Mescheder et al., 2018; Thanh-Tung et al., 2019; Kodali et al., 2017; adopt different forms of gradient penalty terms to further improve training stability. However, it is often observed that such gradient penalty strategy sometimes generate samples with unsatisfying quality, or even do not always converge to the equilibrium point (Mescheder et al., 2018) . In this paper, we propose a general framework named Wasserstein-Bounded GAN (WBGAN), which improve the stability of WGAN training by bounding the Wasserstein term. The highlight is that the instability of WGANs also resides in the dramatic changes of the estimated Wasserstein distance during the initial iterations. Many previous works just focused on improving the gradient penalty term for stable training, while they ignored the bottleneck of the Wasserstein term. The proposed training strategy is able to adaptively enforce the Wasserstein term within a certain value, so as to balance the Wasserstein loss and gradient penalty loss dynamically and make the training process more stable. WBGANs are generalized, which can be instantiated using different kinds of bound estimations, and incorporated into any variant of WGANs to improve the training stability and accelerate the convergence. Specifically, with Sinkhorn distance (Cuturi, 2013; Genevay et al., 2017) for bound estimation, we test three representative variants of WGANs (WGAN-GP (Gulrajani et al., 2017) , WGANdiv (Wu et al., 2018) , and WGAN-GPReal (Mescheder et al., 2018) ) on the CelebA dataset (Liu et al., 2015) . As shown in Fig. 1 This paper introduced a general framework called WBGANs, which can be applied to a variety of WGAN variants to stabilize the training process and improve the performance. We clarify that WBGANs can stabilize the Wasserstein term at the beginning of the iterations, which is beneficial for smoother convergence of WGAN-based methods. We present an instantiated bound estimation method via Sinkhorn distance and give a theoretical analysis on it. It remains an open topic on how to set a better bound for higher resolution image generation tasks.
Propose an improved framework for WGANs and demonstrate its better performance in theory and practice.
{"src": "scitldr-AIC_train", "paper_id": "BkxgrAVFwH"}