Training Generative Adversarial Networks (GANs) is notoriously challenging. We propose and study an architectural modification, self-modulation, which improves GAN performance across different data sets, architectures, losses, regularizers, and hyperparameter settings. Intuitively, self-modulation allows the intermediate feature maps of a generator to change as a function of the input noise vector. While reminiscent of other conditioning techniques, it requires no labeled data. In a large-scale empirical study we observe a relative decrease of 5%-35% in FID. Furthermore, all else being equal, adding this modification to the generator leads to improved performance in 124/144 (86%) of the studied settings. Self-modulation is a simple architectural change that requires no additional parameter tuning, which suggests that it can be applied readily to any GAN. Generative Adversarial Networks (GANs) are a powerful class of generative models successfully applied to a variety of tasks such as image generation BID20 Miyato et al., 2018; Karras et al., 2017) , learned compression BID15 , super-resolution (Ledig et al., 2017) , inpainting (Pathak et al., 2016) , and domain transfer BID13 BID23 .Training GANs is a notoriously challenging task BID6 BID15 as one is searching in a high-dimensional parameter space for a Nash equilibrium of a non-convex game. As a practical remedy one applies (usually a variant of) stochastic gradient descent, which can be unstable and lack guarantees Salimans et al. (2016) . As a result, one of the main research challenges is to stabilize GAN training. Several approaches have been proposed, including varying the underlying divergence between the model and data distributions Mao et al., 2016) , regularization and normalization schemes BID7 Miyato et al., 2018) , optimization schedules (Karras et al., 2017) , and specific neural architectures (Radford et al., 2016; BID21 . A particularly successful approach is based on conditional generation; where the generator (and possibly discriminator) are given side information, for example class labels Mirza & Osindero (2014) ; Odena et al. (2017) ; Miyato & Koyama (2018) . In fact, state-of-the-art conditional GANs inject side information via conditional batch normalization (CBN) layers BID3 Miyato & Koyama, 2018; BID21 . While this approach does help, a major drawback is that it requires external information, such as labels or embeddings, which is not always available.In this work we show that GANs benefit from self-modulation layers in the generator. Our approach is motivated by Feature-wise Linear Modulation in supervised learning (Perez et al., 2018; BID3 , with one key difference: instead of conditioning on external information, we condition on the generator's own input. As self-modulation requires a simple change which is easily applicable to all popular generator architectures, we believe that is a useful addition to the GAN toolbox. We present a generator modification that improves the performance of most GANs. This technique is simple to implement and can be applied to all popular GANs, therefore we believe that selfmodulation is a useful addition to the GAN toolbox.Our results suggest that self-modulation clearly yields performance gains, however, they do not say how this technique results in better models. Interpretation of deep networks is a complex topic, especially for GANs, where the training process is less well understood. Rather than purely speculate, we compute two diagnostic statistics that were proposed recently ignite the discussion of the method's effects.First, we compute the condition number of the generators Jacobian. Odena et al. (2018) provide evidence that better generators have a Jacobian with lower condition number and hence regularize using this quantity. We estimate the generator condition number in the same was as Odena et al. (2018) . We compute the Jacobian (J z ) i,j = δG(z)i δzj at each z in a minibatch, then average the logarithm of the condition numbers computed from each Jacobian.Second, we compute a notion of precision and recall for generative models. Sajjadi et al. (2018) define the quantities, F 8 and F 1/8 , for generators. These quantities relate intuitively to the traditional precision and recall metrics for classification. Generating points which have low probability under the true data distribution is interpreted as a loss in precision, and is penalized by the F 8 score. Failing to generate points that have high probability under the true data distributions is interpreted as a loss in recall, and is penalized by the F 1/8 score. FIG4 shows both statistics. The left hand plot shows the condition number plotted against FID score for each model. We observe that poor models tend to have large condition numbers; the correlation, although noisy, is always positive. This result corroborates the observations in (Odena et al., 2018) . However, we notice an inverse trend in the vicinity of the best models. The cluster of the best models with self-modulation has lower FID, but higher condition number, than the best models without self-modulation. Overall the correlation between FID and condition number is smaller for self-modulated models. This is surprising, it appears that rather than unilaterally reducing the condition number, self-modulation provides some training stability, yielding models with a small range of generator condition numbers.The right-hand plot in FIG4 shows the F 8 and F 1/8 scores. Models in the upper-left quadrant cover true data modes better (higher precision), and models in the lower-right quadrant produce more modes (higher recall). Self-modulated models tend to favor higher recall. This effect is most pronounced on IMAGENET. Overall these diagnostics indicate that self-modulation stabilizes the generator towards favorable conditioning values. It also appears to improve mode coverage. However, these metrics are very new; further development of analysis tools and theoretical study is needed to better disentangle the symptoms and causes of the self-modulation technique, and indeed of others.Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. A ADDITIONAL RESULTS
A simple GAN modification that improves performance across many losses, architectures, regularization schemes, and datasets. 
{"src": "scitldr-AIC_train", "paper_id": "Hkl5aoR5tm"}