In order to alleviate the notorious mode collapse phenomenon in generative adversarial networks (GANs), we propose a novel training method of GANs in which certain fake samples can be reconsidered as real ones during the training process. This strategy can reduce the gradient value that generator receives in the region where gradient exploding happens. We show that the theoretical equilibrium between the generators and discriminations actually can be seldom realized in practice. And this results in an unbalanced generated distribution that deviates from the target one, when fake datepoints overfit to real ones, which explains the non-stability of GANs. We also prove that, by penalizing the difference between discriminator outputs and considering certain fake datapoints as real for adjacent real and fake sample pairs, gradient exploding can be alleviated. Accordingly, a modified GAN training method is proposed with a more stable training process and a better generalization. Experiments on different datasets verify our theoretical analysis. In the past few years, Generative Adversarial Networks (GANs) Goodfellow et al. (2014) have been one of the most popular topics in generative models and achieved great success in generating diverse and high-quality images recently (Brock et al. (2019) ; Karras et al. (2019) ; ). GANs are powerful tools for learning generative models, which can be expressed as a zero-sum game between two neural networks. The generator network produces samples from the arbitrary given distribution, while the adversarial discriminator tries to distinguish between real data and generated data. Meanwhile, the generator network tries to fool the discriminator network by producing plausible samples which are close to real samples. When a final theoretical equilibrium is achieved, discriminator can never distinguish between real and fake data. However, we show that a theoretical equilibrium often can not be achieved with discrete finite samples in datasets during the training process in practice. Although GANs have achieved remarkable progress, numerous researchers have tried to improve the performance of GANs from various aspects ; Nowozin et al. (2016) ; Gulrajani et al. (2017) ; Miyato et al. (2018) ) because of the inherent problem in GAN training, such as unstability and mode collapse. Arora et al. (2017) showed that a theoretical generalization guarantee does not be provided with the original GAN objective and analyzed the generalization capacity of neural network distance. The author argued that for a low capacity discriminator, it can not provide generator enough information to fit the target distribution owing to lack of ability to detect mode collapse. Thanh-Tung et al. (2019) argued that poor generation capacity in GANs comes from discriminators trained on discrete finite datasets resulting in overfitting to real data samples and gradient exploding when generated datapoints approach real ones. As a result, Thanh-Tung et al. (2019) proposed a zero-centered gradient penalty on linear interpolations between real and fake samples (GAN-0GP-interpolation) to improve generalization capability and prevent mode collapse resulted from gradient exploding. Recent work Wu et al. (2019) further studied generalization from a new perspective of privacy protection. In this paper, we focus on mode collapse resulted from gradient exploding studied in Thanh-Tung et al. (2019) and achieve a better generalization with a much more stable training process. Our contributions are as follows: discriminator with sigmoid function in the last layer removed D r = {x 1 , · · · , x n } the set of n real samples D g = {y 1 , · · · , y m } the set of m generated samples D f = {f 1 , · · · , f m } the candidate set of M 1 generated samples to be selected as real D F AR ⊂ {f 1 , · · · , f m } the set of M 0 generated samples considered as real 1. We show that a theoretical equilibrium, when optimal discriminator outputs a constant for both real and generated data, is unachievable for an empirical discriminator during the training process. Due to this fact, it is possible that gradient exploding happens when fake datapoints approach real ones, resulting in an unbalanced generated distribution that deviates from the target one. 2. We show that when generated datapoints are very close to real ones in distance, penalizing the difference between discriminator outputs and considering fake as real can alleviate gradient exploding to prevent overfitting to certain real datapoints. 3. We show that when more fake datapoints are moved towards a single real datapoint, gradients of the generator on fake datapoints very close to the real one can not be reduced, which partly explains the reason of a more serious overfitting phenomenon and an increasingly unbalanced generated distribution. 4. Based on the zero-centered gradient penalty on data samples (GAN-0GP-sample) proposed in Mescheder et al. (2018) , we propose a novel GAN training method by considering some fake samples as real ones according to the discriminator outputs in a training batch to effectively prevent mode collapse. Experiments on synthetic and real world datasets verify that our method can stabilize the training process and achieve a more faithful generated distribution. In the sequel, we use the terminologies of generated samples (datapoints) and fake samples (datapoints) indiscriminately. Tab. 1 lists some key notations used in the rest of the paper. In this paper, we explain the reason that an unbalanced distribution is often generated in GANs training. We show that a theoretical equilibrium for empirical discriminator is unachievable during the training process. We analyze the affection on the gradient that generator receives from discriminator with respect to restriction on difference between discriminator outputs on close real and fake pairs and trick of considering fake as real. Based on the theoretical analysis, we propose a novel GAN training method by considering some fake samples as real ones according to the discriminator outputs in a training batch. Experiments on diverse datasets verify that our method can stabilize the training process and improve the performance by a large margin. For empirical discriminator, it maximizes the following objective: When p g is a discrete uniform distribution on D r , and generated samples in D g are the same with real samples in D r . It is obvious that the discriminator outputs 1 2 to achieve the optimal value when it cannot distinguish fake samples from real ones. For continues distribution p g , Thanh-Tung et al. (2019) has proved that an -optimal discriminator can be constructed as a one hidden layer MLP with O(d x (m + n)) parameters, namely D(x ) ≥ 1 2 + 2 , ∀x ∈ D r and D(y ) ≤ 1 2 − 2 , ∀y ∈ D g , where D r and D g are disjoint with probability 1. In this case, discriminator objective has a larger value than the theoretical optimal version: So the optimal discriminator output on D r and D g is not a constant 1 2 in this case. Even discriminator has much less parameters than O(d x (m + n)), there exists a real datapoint x 0 and a generated datapoint y 0 satisfying D(x 0 ) ≥ 1 2 + 2 and D(y 0 ) ≤ 1 2 − 2 . Whether p g is a discrete distribution only cover part samples in D r or a continues distribution, there exists a generated datapoint y 0 satisfying y 0 ∈ D r . Assume that samples are normalized: Let W 1 ∈ R 2×dx , W 2 ∈ R 2×2 and W 3 ∈ R 2 be the weight matrices, b ∈ R 2 offset vector and k 1 ,k 2 a constant, We can construct needed discriminator as a MLP with two hidden layer containing O(2d x ) parameters. We set weight matrices For any input v ∈ D r ∪ D g , the discriminator output is computed as: where σ(x) = 1 1+e −x is the sigmoid function. Let α = W 1 v − b, we have where l < 1. Let β = σ(k 1 α), we have as k 2 → ∞. Hence, for any input v ∈ D r ∪ D g , discriminator outputs In this case, discriminator objective also has a more optimal value than the theoretical optimal version: So the optimal discriminator output on D r and D g is also not a constant 1 2 in this case.
 We propose a novel GAN training method by considering certain fake samples as real to alleviate mode collapse and stabilize training process.
{"src": "scitldr-AIC_train", "paper_id": "HyxsR24tvS"}